We benchmarked the leading web scraper API services with 12,500 requests across various domains, including eCommerce platforms like Amazon, Target and search engines. Each request was sent real-time and synchronously. See the top web scraping APIs and follow the links to learn more from the vendors:
| Provider | Focus | |
|---|---|---|
1. | Market-leading range of web scraping APIs at cost-effective prices with detailed results | |
2. | Market-leading range of web scraping APIs in e-commerce | |
3. | Market-leading range of web scraping APIs in e-commerce & affordable entry prices | |
4. | All-in-one scraping API solution | |
5. | Market-leading range of web scraping APIs thanks to its community-driven approach | |
6. | Market-leading prices | |
These APIs are for public data. If you are instead looking for APIs that allow web users to collect their or their app users’ private data, you can see examples of web APIs.
The best web scraping APIs & their prices
| Vendors | Starting price (mo) | PAYG plan | Free trial |
|---|---|---|---|
| Bright Data | $500 | ✅ | 7 days |
| Oxylabs | $49 | ❌ | 7 days |
| Smartproxy | $29 | ❌ | 3K free requests |
| Apify | $49 | ✅ | Unlimited |
| Zyte | $100 | ✅ | $5 free credit for a month |
| NetNut | N/A | ❌ | 7 days |
| Nimble | $150 | ❌ | 7 days |
| SOAX | Custom | ❌ | 7 days |
Web scraping API availability
E-commerce APIs
E-commerce APIs are offered by most providers:
| Page type | Bright Data | Oxylabs | Smartproxy | Zyte | Apify | NetNut |
|---|---|---|---|---|---|---|
| Amazon product | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |
| Amazon search | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |
| Target product | ✅ | ❌** | ❌** | ✅ | ❌* | ❌ |
| Walmart product | ✅ | ✅ | ✅ | ❌** | ❌** | ❌ |
| Walmart search | ❌** | ✅ | ✅ | ✅ | ❌** | ❌ |
* Though Apify offers scraping APIs for these page types via its community-maintained APIs, we were not able to access these Actors as part of the plan provided to us by Apify.
** These scrapers exist but their success rate was below our threshold (>90%).
Ranking: Providers are sorted from left to right based on the number of APIs that they offer. If they offer the same number of APIs, they are sorted alphabetically.
For more, see eCommerce-scraping APIs.
Social media APIs
While some providers offer many social media APIs, some don’t offer any:
| Page type | Bright Data | Apify | Smartproxy | NetNut | Oxylabs | Zyte |
|---|---|---|---|---|---|---|
| ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | |
| ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | |
| ✅ | ❌*** | ❌ | ❌ | ❌ | ❌ | |
| TikTok | ✅ | ❌** | ❌ | ❌ | ❌ | ❌ |
| X.com | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
A social network is included with a ✅ only if
- It has an API for all the page types in that social network in our benchmark set and
- Its API has >90% success rate
Learn more about social media scraping and detailed benchmark results.
Search engine APIs
Search engine APIs are offered by all providers:
| Page type | Apify | Bright Data | Smartproxy | Oxylabs | NetNut | Zyte |
|---|---|---|---|---|---|---|
| ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| Bing | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
For more: SERP APIs.
Web scraping API Benchmark results
You can see average number of fields returned by scrapers in 3 categories. The size of each data point represents the number of page types available for scraping for each provider. We also provided median response times. For definitions, see methodology.
How to interpret results?
In scraping, there is a tradeoff between the response time and the amount of data to be retrieved. Since scraping users require fresh data, these services collect data using proxies or unblockers after the client’s request. The more pages that need to be crawled, the longer it takes to return the data.
For example, scraping a search result page on Amazon can either return
- links to products uncovered by search
- Basic data about these products including their pricing.
- A comprehensive dataset about these products including their reviews etc. This will require crawling pages of all products featured in the search page.
The latter approach is what Bright Data’s Amazon Products – Discover by Search URL product follows. As a result, this product’s retrieval time can take significantly longer than other scraping APIs.
Features
Bright Data allows users to specify the data that they will retrieve and receive faster responses with its custom IDE scrapers.
Custom IDE scraper is a way to customize an off-the-shelf web scraping API. Bright Data’s custom IDE module reduced the response time to 3.5 seconds when we reduced the amount of data we requested.
Custom IDE module has ready-to-use templates for commonly used websites (e.g. Amazon, Youtube, Facebook) and also enables the user to modify the templates.
Providers
Bright Data
Best for enterprise-compliant & cost-effective APIs for large-scale use casesBright Data provides more than 230 web scraper APIs, covering a wide range of data sources such as social media, eCommerce, real estate, and travel. Additionally, it offers proxy-based APIs like Web Unlocker and SERP API.
These scraper APIs are equipped with a proxy pool that allows for targeting at both country and city levels across any location. The APIs feature capabilities such as JavaScript rendering, IP rotation, and anti-detection measures.
- Locations: Country and city-level targeting
- Data format: JSON, NDJSON, JSON Lines and CSV
- Price: Scraper APIs start at $500 ($0.85/1K records). Pay as you go $1/1K records.
- Free trial: 7 days for companies (20 free API calls)
Get 25% off Bright Data’s Web Scraping APIs by entering the promo code API25.
Oxylabs
Oxylabs is a premium proxy provider that offers a variety of web scraping APIs, including the Web Scraper API, SERP Scraper API, Real Estate Scraper, and E-Commerce Scraper API.
Their web scraping APIs allow targeting at the country level in 195 locations, while the SERP Scraper API provides more precise targeting options, such as city and coordinate-level targeting. Their Web Scraper API supports headless browser to render and extract data from JavaScript-heavy websites.
- Locations: Country-level targeting
- Data format: HTML, JSON, CSV
- Price: Web Scraper API at $49 for 17,500 results ($2.8/1,000 requests)
- Free trial: 7-day free trial for companies (5,000 requests)
Decodo
Best for flexible scraping API packages & affordable entry priceSmartproxy provides four web scraper API services: Social Media, SERP Scraping, eCommerce, and Web Scraping APIs. These APIs come with a residential proxy network and support country-level targeting, while the SERP Scraping API also enables coordinate-level targeting.
The scraping APIs from Smartproxy include essential features like proxy rotation, anti-detection methods, and JavaScript rendering.
- Locations: Country-level targeting
- Data format: Table and JSON
- Price: $50 Web Scraping API $2.0/1K requests. $30 SERP Scraping API $2.0/1K requests, eCommerce Scraping API $2.0/1K requests.
- Free trial: 7-day free trial (3,000 requests)
Apify
Best for developer-focused web scraping with ready-made solutionsApify is a developer-focused web scraping platform that offers pre-made scrapers and automation tools called Actors. Actors are designed to automate actions or extract data from social media sites, e-commerce platforms, review sites, job portals, and other websites. Every Actor can be accessed via API using Python, JavaScript, or HTTP requests.
You can use Actors as they are, ask to modify them for your use case, or create your own. Developers can create and run Actors in various programming languages (such as JavaScript/TypeScript and Python) by using code templates, universal scrapers, or the open-source web scraping library, Crawlee.
Apify runs on a cloud-based infrastructure with built-in scraping features such as datacenter and residential proxies, automatic IP rotation, CAPTCHA solving, monitoring, scheduling, and integrations.
- Locations: Country-level targeting
- Data format: CSV, JSON, Excel, XML, HTML, RSS
- Price: Monthly plans starting from $49
- Free trial: A free account comes with $5 platform credits
Zyte
Best for users looking for a all-in-one scraper API solutionZyte provides a general-purpose scraper API with proxy management features and browser automation capabilities. The scraper API allows for handling request headers, cookies, and toggling JavaScript.
- Locations: Country level targeting ( 19 countries)
- Data format: JSON or HTML
- Price: $100
- Free trial: $5 credit
Nimble
All-in-one API solution including unblocker, residential, and APINimble provides offers general purpose, SERP, e-commerce, and maps APIs featuring integrated rotating residential proxies and unlocker proxy solutions. The Web API is capable of handling batch requests, allowing up to 1,000 URLs in each batch. The Nimble Web API offers three methods for data delivery:
- Real-time: Data is collected and instantly returned to the user.
- Cloud Storage: Collected data is sent to the user’s chosen cloud storage service.
- Push/Pull: Data is stored on Nimble’s servers and can be accessed through a provided URL for download.
- Locations: Country and city level targeting
- Data format: HTML and JSON
- Price: $150 ($7.5/GB residential, $2.6/CPM API)
- Free trial: 7-day trial
NetNut
NetNut provides SERP Scraper (Google) and LinkedIn API, offering customization options to modify requests based on various parameters such as geographic location, pagination, and localization preferences including language and country settings.
- Locations: Country level targeting
- Data format: JSON or HTML
- Price: Custom
- Free trial: 7-day trial
SOAX
SOAX offers social media, SERP and and eCommerce APIs. The vendor provides built-in proxy management capabilities and handles pagination. Users can set a max page parameter for scraping multiple pages.
- Locations: Country level targeting
- Data format: JSON or HTML
- Price: Custom
- Free trial: 7-day trial
Web scraping APIs: Benchmark methodology
Test URLs
We used 3 URL groups in testing:
- 1,700 eCommerce URLs as explained in e-commerce scraper benchmark.
- 1,100 social media URLs as outlined in social media scraper benchmark.
- 200 query result pages as explained in SERP API benchmark.
Speed & latency
- Proxies and web unblocker: Response time is measured.
- Scraping API: Response time is calculated as the difference between webhook callback time and request time.
To be fair, all providers’ response times are calculated in the same set of pages where they all returned successful responses. It would not be fair to compare the response time of an unsuccessful response to a successful one since an unsuccessful response can potentially be generated much faster.
For example, if four unblockers were run on 600 URLs and they all returned successful results for only 540 URLs, these 540 URLs form the basis of response time calculation.
Success rates
Requirements for a successful request for a web scraper API:
- HTTP response code: 200
- A response longer than 500 characters
If a web scraper returns successful results more than 90% of the time for a specific type of page (e.g. Walmart search pages) and if the correctness of the results is validated by random sampling of 10 URLs, then we list that provider as a scraping API provider for that type of page.
Most scraper APIs had more than 90% success rates for their target pages. Therefore, rather than focusing on 1-2% differences between different APIs, we list all APIs that returned successful results more than 90% of the time.
Even though we used fresh URLs, a small percentage of URLs were discovered to return 404 during the test. They were excluded from the test.
Determining participants
- Web scraper APIs: Participants’ websites were scanned to identify relevant scrapers.
- Proxies: All providers except Zyte were included.
Average # of fields
- For each successful API result, we count the number fields returned in the JSON file. Each key is counted regardless of its value.
What is a web scraping API?
An API is a tool that facilitates communication between clients and web servers, allowing data exchange rather than data collection. If a website supports API functionality, it can be used to retrieve data. APIs fall into these categories:
- Scraping APIs: Hosted by web data infrastructure providers, they convert publicly available web pages into structured data.
- Other public APIs: Hosted by external providers like Facebook or Google, these APIs are available to developers and can be free or require registration.
- Internal APIs: Restricted to use within an organization, these private APIs enable secure internal communication and data transfers. Regardless of the type, APIs can pose security risks if not properly managed.
4 examples of web APIs
These are example public APIs that include web data:
1. Amazon Product Advertising API
This API gives users access to Amazon’s product data, including customer reviews and product information. To use it, users must register for the Product Advertising API and have an Amazon Associates account. Through HTTP client requests with specified parameters like keywords or brands, users can retrieve details for up to 10 products per search.
Once you have registered the Product Advertising API, you can send requests to the API using an HTTP client. You can search for a specific item on Amazon by requesting parameters supported by the API, such as keywords, titles, and brands.
Figure 1: Product output obtained from the Amazon product page using an API request.
2. Google Analytics Data API
Google Analytics Data API v4 provides free access to GA4 data, allowing users to display and collect report data. The API supports generating reports like pivot tables, cohort analysis, and lifetime value reports, helping users analyze their website’s performance.
3. Twitter API
The Twitter API allows developers to extract Twitter data, subject to rate limits, after signing up for a developer account. Businesses can use the API to analyze historical and real-time data. For example, LikeFolio uses the API to analyze social data and predict shifts in consumer trends, as demonstrated by its use of Twitter data to inform investment strategies. 2
The chart below compares the stock price in gray to the disparity in consumer mentions of Roku goods and services in green.
Figure 2: The difference between the stock price and consumer demand for the company’s goods and services.
Source: Twitter Developer Platform
4. Instagram API
The Instagram Basic Display API enables businesses and creators to access publicly available Instagram data, such as profiles, images, and hashtags. However, the API imposes restrictions on data access, like limiting hashtag requests to 30 per week. Access requires a Facebook developer account and a registered website.
Figure 3: An example of obtaining image data from Instagram using API
Source: Meta
How do web scraping APIs work?
Web APIs can be accessed by client devices such as phones and laptops to access the data.
Assume a client types a URL into the web browser search box;
- The URL is in HTTP format, such as “https://aimultiple.com”.
- The client sends an API request to the target web server to gain access to the needed data.
- HTTP API request specifies the search item using the “get” verb.
- The API receives the API request from the client in the form of an HTTP request and returns the requested information for each search request based on the item attributes specified in the “get” request.
- The API will then respond to the client’s specific request, usually in the form of JavaScript Object Notation (JSON) or Extensible Markup Language (XML).
The web API generates a data pipeline between the client’s device and the target web server to exchange data using the HTTP protocol. Both request (client) and response (web server) have HTTP headers.
HTTP headers provide additional context about the request or response for the client and web server to communicate, such as “Content-Type”, “Content-Location”, “User-Agent”, or “Accept-Ranges” (see Figure 4).
Figure 4: An example of an HTTP request header containing a few pieces of information
Data extraction with APIs: why and when should you use APIs?
There are a few common methods for obtaining data, including pre-packaged datasets, collecting your data, or data from outside sources. Either way, when it comes to data collection, you will need a tool to handle data collection issues.
Web scraping tools and APIs enable businesses to collect data from internal and external sources. However, they have some differences regarding technical effort, cost, and data accessibility.
The technical difficulty of web scrapers varies depending on whether you use in-house or outsourced web scrapers. However, web scraping tools are less flexible than code-based web scraping solutions such as scraping APIs.
If you have basic programming knowledge and do not have a budget for pre-built web scraping solutions, you can use APIs for your data collection projects.
However, the website from which you want the data must provide the API technology if you want to use APIs for data collection. Otherwise, APIs cannot be an option to collect data.
APIs, such as the Twitter API, are provided by the website from which you require the data. Because the website’s API provides the data, requesters have authorized access to it. You need not be concerned about being identified as a malicious actor. You must, however, follow the terms and conditions outlined in their API guideline.
Key features to consider while choosing web scraping API
1. Javascript rendering
Websites collect data and provide tailored content based on visitor activities using various tracking techniques such as browser fingerprinting, cookies, and web beacons. Every time a user visits a website, the content changes.
Dynamic websites use client-side scripting to change the content based on users’ input and behaviors, such as resizing images according to clients’ screens or displaying website content based on visitor country.
For example, when you make a connection request to an API to access the target website’s data, the API receives your request and returns the requested information. Unless you use a headless browser, the target web server and the websites’ API can access information about your device, such as your machine IP address, browser type, geolocation, and preferred language along the way (see Figure 5).
Javascript rendering handles the parsing of the HTML and the CSS documents and images on the requested page and displays the parsed content on the clients’ browser screen.
To render dynamic website contents, you need to make an HTML request to the target website and invoke the render function to run Javascript code in the background to display the web content on your browser. If you do not want to deal with dynamic web page rendering, you should look for a scraping API that supports javascript rendering.
Figure 5: Example of a browser fingerprinting
2. Unlimited bandwidth
The maximum rate of data transfer in a computer network is referred to as bandwidth. The amount of data you need to collect should be balanced by your bandwidth rate. Otherwise, the data transferred from the web server to your machine will exceed the maximum data transfer rate, causing the bandwidth to be throttled. Unlimited bandwidth allows businesses to:
- Manage data traffic in the network
- Keep data speed under control and allow for much faster data transmission than a constrained bandwidth rate
- Receive large amounts of data from other servers without bandwidth throttling
3. CAPTCHA & Anti-Bot Detection
Websites employ various anti-scraping techniques such as Robots.txt, IP blockers, and CAPTCHA to manage the connection request to their websites and protect their content from a specific type of attack such as bots.
CAPTCHA (Completely Automated Public Turing Test to Tell Computers and Humans Apart) is an anti-scraping method used by web services such as Google to prevent unauthorized users from accessing web data. CAPTCHAs are programmed to be unreadable by machines. Websites use CAPTCHA technologies to distinguish between human and malicious bot activities. There are three types of CAPTCHAs:
To learn how to bypass CAPTCHA challenges, check out “The Ultimate Guide to Avoiding CAPTCHAs in Web Scraping“
- Text-based: This CAPTCHA type requires users to retype distorted words and numbers they see in a given image (see Figure 6). The provided text is unrecognizable by bots.
Figure 6: An example of text-based CAPTCHA
- Image-based: Image-based CAPTCHAs use object detection and target recognition technologies. The user is asked to select specific objects from a set of images the website provides (see Figure 7).
Figure 7: An example of image-based CAPTCHA
- Audio-based: When you click the audio icon in a distorted image, it will speak the letter in the image for you while making some gibberish noises to prevent bots.
If the target website from which you need data has its API, you do not need to be concerned about the legality of data scraping and being detected by the website. However, if you use a third-party scraping API solution, you need to either overcome the captcha yourself or outsource the captcha solving to the service provider.
Check out top 7 web scraping best practices to learn more on how to overcome web scraping challenges.
4. Auto Parsing
After data extraction, the collected data may be structured, semi-structured, or unstructured as it is gathered from various data sources. To extract value from collected data, you must parse the extracted data to convert it into a more readable format.
You can build your parser or leverage outsourced data parsing tools to convert extracted data into the desired data format. However, in-house data parsing tools might have additional overhead costs. Outsourcing the development and maintenance of data parsing infrastructure will allow you to focus on data analysis.
5. Geotargeting
Websites block or restrict access to specific content based on users’ geolocation for various reasons, including fraud prevention, price discrimination, and malicious traffic blocking. Scraping APIs enable users to access geo-targeted content in order to provide localized information.
6. Automatic Proxy Rotation
The crawl rate limit is an anti-scraping technique that websites use to manage the volume of requests to their websites. When a client repeatedly requests a web server’s API from the same IP address, the website recognizes the client as a bot and restricts access to the web content. Automatic proxy rotation enables clients to change their IP addresses for each connection request.
Comments
Your email address will not be published. All fields are required.