How to Scrape X.com (Twitter) with Python and Playwright

We used Python and Playwright to test Twitter (X) data collection methods focusing on most prominent pages:

Twitter scraping methodology

We have performed all tests without logging in. Our aim is to collect public data.

Pages to be scraped

1. User profiles to get bio text, follower numbers, and join date (i.e. date when the user joined X.com)
2. Hashtag pages to see tweets under a specific hashtag
3. Search result pages to find tweets based on keywords

Scraping methods

A combination of web automation tools and network configurations was used to manage dynamic content and bypass anti-bot protections:

1. Python + Playwright (sync API): We used Playwright’s synchronous API version, suitable for JavaScript-heavy sites like Twitter.
2. Chromium browser (non-headless mode): Twitter’s anti-bot systems are more likely to block headless sessions.
3. Proxy configurations: We tested three network setups:
   • No proxy: We used the service with our internet connection without relying on any proxies. This is not a scalable approach.
   • Residential proxy
   • Web unblocker.

Twitter web scraping results by proxy configuration

While scraping Twitter data, we tried to get data like the bio, follower count, and join date by inspecting the page’s raw HTML using methods like page.content() or checking network responses. But we noticed that some information, like follower counts, was missing. These parts of the page are loaded later with JavaScript.

We used CSS selectors to target the data from the fully loaded (rendered) page to solve this. This approach is often necessary when scraping Twitter, but not all elements are equally easy to grab. Some change more often or load slowly.

Using a tool like unblocker helped improve consistency by ensuring the page loads and behaves similarly to how it would for a real user, reducing errors and making the CSS selectors more reliable.

CSS selector reliability on profile pages

Even if your Twitter scraper gets through the platform’s proxy detection methods, it may fail because the data is no longer where you expect it to be. We tested the reliability of different selectors used to extract common profile fields:

1. Profile page scraping

We extracted four common data fields from Twitter profile pages: bio, follower count, following count, and join date. Below is a summary of how each proxy configuration performed:

2. Hashtag page scraping

Unlike profile pages, Twitter hashtag feeds are harder to scrape reliably. When accessing these pages without a proxy, we were often redirected or faced CAPTCHA challenges. Residential proxies performed slightly better, allowing the page to begin loading, but they still failed to deliver usable data consistently.

The only method that successfully rendered and accessed the hashtag feed was the unblocker.

*Represents a partial result. The hashtag page began loading, but the tweet data didn’t appear.

3. Search page scraping

Search result pages on Twitter are the most difficult to access via web scraping, as they include rate-limiting, SSL certificate checks, and dynamic behavior that blocks web scraping tools.

• Connections without proxies failed.
• Residential proxies had slightly better page load performance, but the data rarely became available. Attempts often ended in timeouts or HTTP 502 errors.
• Even unblocker, which successfully handled other Twitter endpoints, failed here. It triggered certificate errors (like ERR_CERT_AUTHORITY_INVALID) and inconsistent server responses.

Technical challenges in Twitter web scraping

We encountered various recurring errors during our scraping tests, especially when accessing more protected pages like search results. These issues often stemmed from SSL problems, slow-loading pages, or aggressive anti-bot protections on Twitter.

Here is a breakdown of the most common errors and what likely caused them:

• ERR_CERT_AUTHORITY_INVALID: This error points to SSL certificate issues, which are often caused by misconfigured proxies or when using advanced proxy tools like Unblocker, which inject their certificates.
• Timeout 30000ms exceeded: A common Playwright error indicating that the page took too long to load. This typically happens due to heavy JavaScript rendering or slow proxy connections that delay full page hydration.
• 502 Server Error or read timeout: These errors suggest the server blocked or dropped the request, especially when accessing search pages. Twitter may be actively denying access to automated traffic.
• Event loop is closed!: This is a Playwright or asyncio-related error that usually occurs after a crash, an abrupt disconnection, or an incomplete async response. It often requires resetting the browser context or reinitializing the session.

Twitter web scraping findings

While specific errors were more technical (e.g., timeouts, SSL issues), some challenges were inherent to each Twitter page type’s structure and protection level. After testing multiple proxy setups, we found clear differences in reliability across profile, hashtag, and search pages:

• Unblocker is the most reliable option for scraping profile pages, especially when retrieving follower/following counts.
• Hashtag and search result pages remain difficult to scrape with any method tested.
• Running a visible (headful) browser helps reduce bot detection, but it is not enough.
• JavaScript rendering and hydration delays (i.e., the time it takes for data to appear on the page fully) significantly affect scraping accuracy.

FAQ about web scraping Twitter

External Links

• 1. Twitter API Documentation | Docs | Twitter Developer Platform .