scrapy-impersonate

scrapy-impersonate is a Scrapy download handler. This project integrates curl_cffi to perform HTTP requests, so it can impersonate browsers' TLS signatures or JA3 fingerprints.

Installation

pip install scrapy-impersonate

Activation

To use this package, replace the default http and https Download Handlers by updating the DOWNLOAD_HANDLERS setting:

DOWNLOAD_HANDLERS = {
    "http": "scrapy_impersonate.ImpersonateDownloadHandler",
    "https": "scrapy_impersonate.ImpersonateDownloadHandler",
}

By setting USER_AGENT = None, curl_cffi will automatically choose the appropriate User-Agent based on the impersonated browser:

USER_AGENT = ""

Also, be sure to install the asyncio-based Twisted reactor for proper asynchronous execution:

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

Usage

Set the impersonate Request.meta key to download a request using curl_cffi:

import scrapy
class ImpersonateSpider(scrapy.Spider):
    name = "impersonate_spider"
    custom_settings = {
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "USER_AGENT": "",
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_impersonate.ImpersonateDownloadHandler",
            "https": "scrapy_impersonate.ImpersonateDownloadHandler",
        },
        "DOWNLOADER_MIDDLEWARES": {
            "scrapy_impersonate.RandomBrowserMiddleware": 1000,
        },
    }
def start_requests(self):
        for _ in range(5):
            yield scrapy.Request(
                "https://tls.browserleaks.com/json",
                dont_filter=True,
            )
def parse(self, response):
        # ja3_hash: 98cc085d47985d3cca9ec1415bbbf0d1 (chrome133a)
        # ja3_hash: 2d692a4485ca2f5f2b10ecb2d2909ad3 (firefox133)
        # ja3_hash: c11ab92a9db8107e2a0b0486f35b80b9 (chrome124)
        # ja3_hash: 773906b0efdefa24a7f2b8eb6985bf37 (safari15_5)
        # ja3_hash: cd08e31494f9531f560d64c695473da9 (chrome99_android)
yield {"ja3_hash": response.json()["ja3_hash"]}

impersonate-args

You can pass any necessary arguments to curl_cffi through impersonate_args. For example:

yield scrapy.Request(
    "https://tls.browserleaks.com/json",
    dont_filter=True,
    meta={
        "impersonate": browser,
        "impersonate_args": {
            "verify": False,
            "timeout": 10,
        },
    },
)

Some arguments worth knowing about:

| Argument | Description | | --- | --- | | http_version | Set to "v3" to use HTTP/3. Targets with an HTTP/3 fingerprint are marked in the table below | | doh_url | Resolve DNS over HTTPS instead of using the system resolver (curl_cffi >= 0.16.0) | | dns | DNS server address, or list of addresses, to resolve with instead of the system resolver. Requires a curl_cffi build with c-ares (curl_cffi >= 0.17.0) | | interface | Bind the request to a network interface or local source address | | extra_fp | Fine-tune fingerprint details on top of the impersonated browser |

[!WARNING]

allow_redirects is forced to False so that redirects are handled by Scrapy. Re-enabling it through impersonate_args makes curl_cffi follow redirects internally, which bypasses Scrapy's redirect and offsite middlewares and exposes you to redirect-based SSRF.

curl-options

For settings that have no curl_cffi argument, impersonate_curl_options passes raw libcurl options through. Keys can be CurlOpt members or their names, and they take precedence over the options set by this handler.

The most common use is pinning the header order, which browsers keep stable and WAFs check (HTTPHEADER_ORDER requires curl_cffi >= 0.16.0):

yield scrapy.Request(
    "https://tls.browserleaks.com/json",
    dont_filter=True,
    meta={
        "impersonate": "chrome",
        "impersonate_curl_options": {
            "HTTPHEADER_ORDER": "Host,Connection,User-Agent,Accept,Referer",
        },
    },
)

Supported browsers

The following browsers can be impersonated (curl_cffi >= 0.17.0):

| Browser | Version | OS | Name | HTTP/3 | | --- | --- | --- | --- | --- | | !Chrome | 99 | Windows 10 | chrome99 | | | !Chrome | 99 | Android 12 | chrome99_android | | | !Chrome | 100 | Windows 10 | chrome100 | | | !Chrome | 101 | Windows 10 | chrome101 | | | !Chrome | 104 | Windows 10 | chrome104 | | | !Chrome | 107 | Windows 10 | chrome107 | | | !Chrome | 110 | Windows 10 | chrome110 | | | !Chrome | 116 | Windows 10 | chrome116 | | | !Chrome | 119 | macOS Sonoma | chrome119 | | | !Chrome | 120 | macOS Sonoma | chrome120 | | | !Chrome | 123 | macOS Sonoma | chrome123 | | | !Chrome | 124 | macOS Sonoma | chrome124 | | | !Chrome | 131 | macOS Sonoma | chrome131 | | | !Chrome | 131 | Android 14 | chrome131_android | | | !Chrome | 133 | macOS Sequoia | chrome133a | | | !Chrome | 136 | macOS Sequoia | chrome136 | | | !Chrome | 142 | macOS Tahoe | chrome142 | | | !Chrome | 145 | macOS Tahoe | chrome145 | ✅ | | !Chrome | 146 | macOS Tahoe | chrome146 | ✅ | | !Chrome | 150 | macOS Tahoe | chrome150 | ✅ | | !Chrome | 151 | Windows 10 | chrome151 | ✅ | | !Chrome | 151 | macOS 26.0 | chrome151_macos | ✅ | | !Chrome | 152 | Windows 10 | chrome152 | ✅ | | !Chrome | 152 | macOS 26.0 | chrome152_macos | ✅ | | !Edge | 99 | Windows 10 | edge99 | | | !Edge | 101 | Windows 10 | edge101 | | | !Safari | 15.3 | macOS Big Sur | safari153 | | | !Safari | 15.5 | macOS Monterey | safari155 | | | !Safari | 17.0 | macOS Sonoma | safari170 | | | !Safari | 17.2 | iOS 17.2 | safari172_ios | | | !Safari | 18.0 | macOS Sequoia | safari180 | | | !Safari | 18.0 | iOS 18.0 | safari180_ios | | | !Safari | 18.4 | macOS Sequoia | safari184 | | | !Safari | 18.4 | iOS 18.4 | safari184_ios | | | !Safari | 26.0 | macOS Tahoe | safari260 | | | !Safari | 26.0 | iOS 26.0 | safari260_ios | | | !Safari | 26.0.1 | macOS Tahoe | safari2601 | | | !Safari | 27.0 | macOS 27.0 | safari270 | ✅ | | !Safari | 27.0 | iOS 27.0 | safari270_ios | ✅ | | !Firefox | 133.0 | macOS Sonoma | firefox133 | | | !Firefox | 135.0 | macOS Sonoma | firefox135 | | | !Firefox | 144.0 | macOS Tahoe | firefox144 | | | !Firefox | 147.0 | macOS Tahoe | firefox147 | ✅ | | !Firefox | 153.0 | Windows 10 | firefox153 | | | !Firefox | 154.0 | Windows 10 | firefox154 | | | !Firefox | 155.0 | Windows 10 | firefox155 | | | !Tor | 14.5 | macOS Sonoma | tor145 | |

_Notes:

1. The old Safari target names (safari15_3, safari15_5, safari17_0, safari17_2_ios, safari18_0, safari18_0_ios) are kept as deprecated aliases. Prefer the new names (safari153, safari155, …) for new code.
2. You can also pass a floating value like impersonate="chrome", "safari", "safari_ios" or "firefox" to let curl_cffi pick the most recent fingerprint without pinning a version. "chrome" resolves to the Windows profile (chrome152); use the _macos targets for Chrome on macOS.
3. RandomBrowserMiddleware rotates across chrome, firefox, safari, edge and tor by default. Override with IMPERSONATE_BROWSERS to narrow the set (e.g. IMPERSONATE_BROWSERS = ["chrome", "firefox"]).

Development

Install the development dependencies and run the test suite:

pip install -r requirements-dev.txt
pip install -e .
pytest

The tests spin up local HTTP/HTTPS servers and a CONNECT proxy, so no network access is required.

Thanks

This project is inspired by the following projects:

+ curl_cffi - Python binding for curl-impersonate via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints. + curl-impersonate - A special build of curl that can impersonate Chrome & Firefox + scrapy-playwright - Playwright integration for Scrapy