Solution
Keep a crawl running when a challenge appears mid-job. Detect the type, post one task, get the token and carry on without a headful browser farm.
Who this is for
Python and Node engineers who own the crawlers behind a data product, plus data-platform teams at scraping agencies. Their week goes to parser drift, proxy rotation and queue depth, not to challenge pages. The crawler is usually a Scrapy or Playwright fleet on a schedule, feeding a warehouse that someone downstream reads every morning.
The problem
A crawler that runs clean at 10k pages starts returning Turnstile interstitials and five-second challenge pages at 100k. The bottleneck moves from bandwidth to challenge throughput: the queue backs up, retries burn proxy budget, and the nightly job misses its window. By the time anyone looks, the warehouse holds a partial day.
Works with
[ 04 ]Questions about this one
For most types, no. You send the sitekey and the page URL over HTTP and get back a token to post with the form. A browser only helps when the challenge issues a cookie rather than a form token, as the Cloudflare interstitial does.
Match on the response body and headers: a cf-turnstile container, a g-recaptcha div carrying data-sitekey, a 403 with a DataDome cookie. Each type page lists the markers it is identified by.
Concurrency is an account-level setting rather than a per-type quota, and is raised from the dashboard. Past the ceiling the API queues tasks instead of rejecting them, so a burst does not lose work.
No. A task that returns no token is not billed. Budget from your solve count rather than your attempt count.
Create a key, paste it into the sample above, and watch a real token come back. No card, no sales call.
