How to Handle CAPTCHAs When Web Scraping

Handling CAPTCHAs when web scraping is one of the most frustrating challenges developers face — but with the right strategy, you can reliably bypass reCAPTCHA, hCaptcha, and Cloudflare Turnstile at scale using a scraping API.

1. What is a CAPTCHA and why do sites use them?

CAPTCHA stands for Completely Automated Public Turing test to tell Computers and Humans Apart. Websites use CAPTCHAs as a first line of defense against automated bots — including scrapers, credential stuffers, and spam bots. However, CAPTCHAs also block legitimate data collection workflows, which is why solving them programmatically is a core skill for any serious scraper.

In practice, a CAPTCHA challenge is triggered when a site detects suspicious behavior: too many requests from the same IP, missing browser headers, or JavaScript that does not execute. Therefore, understanding why a CAPTCHA fires is just as important as knowing how to solve it.

2. Types of CAPTCHAs you will encounter

Not all CAPTCHAs are equal. As a result, the solving strategy differs depending on which one you face. Here are the most common types:

• reCAPTCHA v2 — The classic “I am not a robot” checkbox, sometimes followed by image grids.
• reCAPTCHA v3 — Invisible; assigns a risk score based on user behavior. No visible challenge, but a low score blocks the request.
• hCaptcha — A privacy-focused alternative to reCAPTCHA, widely used on Cloudflare-protected sites.
• Cloudflare Turnstile — Cloudflare's modern CAPTCHA replacement; relies on browser fingerprinting and JS challenges.
• Text/image CAPTCHAs — Older distorted-text or image-recognition challenges, still found on legacy sites.
• FunCaptcha (Arkose Labs) — Interactive puzzle challenges used by large platforms like Roblox and Outlook.

3. Why CAPTCHAs are hard to bypass with plain requests

A naive HTTP request with requests or axios will almost always trigger a CAPTCHA on modern sites. This is because these libraries do not execute JavaScript, do not send realistic browser fingerprints, and reuse the same IP address repeatedly. For example, Cloudflare's Bot Management checks TLS fingerprints, HTTP/2 frame ordering, and browser behavior — none of which plain HTTP clients replicate.

Moreover, even headless browsers like Puppeteer or Playwright are increasingly detected via their automation-specific navigator properties. In addition, maintaining a pool of residential proxies and rotating them correctly adds significant operational complexity. This is precisely why using a dedicated scraping API is the most practical solution for production-grade workflows.

4. Solving CAPTCHAs automatically with ScrapingBot

ScrapingBot's API handles CAPTCHA solving transparently. Instead of managing your own headless browser fleet, proxy rotation, and CAPTCHA-solving services, you send a single API request and receive the rendered HTML. ScrapingBot internally uses a real browser, rotates residential IPs, and resolves CAPTCHA challenges before returning the page content.

There are two relevant endpoints depending on your use case:

• Raw Scraper — For static or lightly protected pages. Fast and cost-effective.
• Real Browser Scraper — Launches a full Chromium instance with JavaScript execution. Required for reCAPTCHA v3, hCaptcha, and Cloudflare Turnstile.

Furthermore, ScrapingBot automatically retries failed requests and rotates proxies, so you do not need to implement retry logic yourself.

5. Code examples: Python and Node.js

Python — Real Browser Scraper

Use this when the target page is protected by reCAPTCHA v3 or Cloudflare Turnstile. The realBrowser parameter tells ScrapingBot to use a full JS-rendering engine:

import requests
from requests.auth import HTTPBasicAuth

API_USER = 'your_username'
API_KEY  = 'your_api_key'
TARGET   = 'https://example.com/protected-page'

url = 'https://api.scraping-bot.io/scrape/raw-html'
payload = {'url': TARGET, 'options': {'realBrowser': True}}

response = requests.post(
    url,
    json=payload,
    auth=HTTPBasicAuth(API_USER, API_KEY)
)

print(response.status_code)
print(response.text[:500])

Python — Polling for async results

For heavy pages, ScrapingBot processes requests asynchronously. As a result, you need to poll the response endpoint until the status is finished:

import requests, time
from requests.auth import HTTPBasicAuth

API_USER = 'your_username'
API_KEY  = 'your_api_key'
auth     = HTTPBasicAuth(API_USER, API_KEY)

# Step 1: submit the job
job = requests.post(
    'https://api.scraping-bot.io/scrape/raw-html',
    json={'url': 'https://example.com', 'options': {'realBrowser': True}},
    auth=auth
).json()

response_id = job['responseId']

# Step 2: poll until done
while True:
    result = requests.get(
        f'https://api.scraping-bot.io/scrape/raw-html/response/{response_id}',
        auth=auth
    )
    if result.status_code == 200:
        print(result.text[:500])
        break
    time.sleep(3)

Node.js — Fetch with async/await

const fetch = require('node-fetch');
const { Buffer } = require('buffer');

const API_USER = 'your_username';
const API_KEY  = 'your_api_key';
const auth     = 'Basic ' + Buffer.from(`${API_USER}:${API_KEY}`).toString('base64');

async function scrapeWithCaptcha(targetUrl) {
  const res = await fetch('https://api.scraping-bot.io/scrape/raw-html', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json', 'Authorization': auth },
    body: JSON.stringify({ url: targetUrl, options: { realBrowser: true } })
  });
  const data = await res.json();
  console.log(data);
}

scrapeWithCaptcha('https://example.com/protected');

6. Best practices to avoid triggering CAPTCHAs

The best way to handle CAPTCHAs is to avoid triggering them in the first place. However, this is not always possible, so combine both prevention and solving strategies:

• Rotate IP addresses — Use residential or datacenter proxies. ScrapingBot handles this automatically.
• Throttle your requests — Add random delays between requests (1–5 seconds) to mimic human browsing.
• Send realistic headers — Include User-Agent, Accept-Language, Referer, and Accept-Encoding. ScrapingBot injects these by default.
• Maintain session cookies — Reuse cookies across requests on the same domain. CAPTCHAs are less likely for “known” sessions.
• Avoid scraping during peak hours — Some sites increase bot detection sensitivity during high-traffic periods.
• Use the Real Browser endpoint for JS-heavy sites — Plain HTTP scraping on a JS-rendered page always looks like a bot.

7. CAPTCHA-solving approaches: comparison table

8. Going further

Handling CAPTCHAs when web scraping does not have to be a roadblock. In summary, the most reliable approach is to combine CAPTCHA avoidance best practices — realistic headers, request throttling, session reuse — with a scraping API that solves challenges transparently. ScrapingBot's Real Browser endpoint takes care of reCAPTCHA, hCaptcha, and Cloudflare Turnstile out of the box, so you can focus on parsing the data you actually need.