TRAWL

Self-hosted web scraping for applications and AI agents. Get page content through an API or MCP, with built-in handling for supported browser challenges and CAPTCHAs.

TRAWL starts with HTTP, reuses verified clearance and opens Camoufox when a browser is needed. Repeat requests fetch fresh content without repeating a full solve whenever the site accepts the saved session.

Website · Documentation · Latest release

Features

• Challenge handling: Cloudflare, Anubis, Akamai, Imperva, DDoS-Guard and supported AWS WAF and DataDome flows. Built-in CAPTCHA paths include Turnstile, reCAPTCHA v2, hCaptcha, GeeTest Slide, ALTCHA, Friendly Captcha and CAP. Coverage and limits.
• Fast repeat requests: automatic HTTP clearance reuse, Redis or in-memory caching, and browser fallback when required. No named session is needed for shared clearance reuse.
• MCP for agents: read Markdown, scrape HTML, extract structured JSON, capture screenshots and inspect browser activity from known URLs.
• Isolated browser sessions: retain cookies, localStorage, sessionStorage and browser identity across native API, MCP and FlareSolverr-compatible requests.
• API and forward proxy: use /scrape, the FlareSolverr-compatible /v1 endpoint, or a challenge-aware HTTP/HTTPS proxy with WebSockets and Range support.
• Local dashboard: persistent request history, tier outcomes, failure causes and live updates.
• Local solving by default: optional 2Captcha fallback and opt-in Firefox TLS/HTTP2 impersonation; Bun fetch remains the default HTTP transport.

Sponsors

View/Collapse All

Bright Data - The most powerful platform for Web Unlocker, SERP API and web scraping tools. Why Bright Data? • Web Unlocker - bypass any anti-bot protection • SERP API - real-time Google, Bing & more results • Scraping Browser & dedicated scrapers • Massive residential proxy network • Built for scale and reliability Get started for free with Bright Data!

NodeMaven - The most efficient proxy provider for Web Scrapping and Automation with the Highest Quality IP on the market. Why NodeMaven? • ZIP targeting • 99.9% uptime • IP filtering: all proxies have fraud score <97% • No KYC required • Unique free tools: Proxy Bandwidth Checker, Meta Tag Checker, IP Lookup and others! Special codes for TRAWL users: • TRAWL35 - 35% off to Mobile and Residential Proxies • TRAWL40 - 40% off to ISP (Static) Proxies

Performance

Selected same-machine comparisons show faster first and repeat responses than FlareSolverr and Byparr on the default Cloudflare workload:

| Protected page | Best first | Best repeat | | ----------------------- | ---------: | ----------: | | NopeCHA Cloudflare demo | 6.0 s | 879 ms | | 1337x | 3.8 s | 85 ms | | EZTV | 4.6 s | 473 ms |

Measured 2026-10-17 on a development build after v1.8.0. These are verified minima from the recorded sample. First requests used a warm browser with an empty target cache; repeats are fresh fetches with reusable clearance. Results vary by site, IP and session state.

Full comparison, medians and CPU/RAM · Method · Performance guide

Quick start

Run one browser with MCP, an in-memory clearance cache and persistent dashboard history:

docker run -d --name trawl \
  -p 127.0.0.1:8191:8191 \
  --shm-size=1gb \
  -e SESSION_CACHE_DRIVER=memory \
  -e BROWSER_POOL_SIZE=1 \
  -e MCP_ENABLED=true \
  -e METRICS_DASHBOARD_ENABLED=true \
  -v trawl_metrics:/data/metrics \
  ghcr.io/germondai/trawl:1.9.0
curl http://localhost:8191/health

Open the local dashboard. Keep the API and MCP endpoint on a trusted network; dashboard history survives container recreation through the mounted volume.

For Redis and Compose deployments, see the quick-start guide and Compose examples. Use :1.9.0-baseline on older CPUs or NAS hardware.

API

Native scraping

curl -X POST http://localhost:8191/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://nopecha.com/demo/cloudflare","solveEmbeddedCaptchas":false}'

Returns HTML, cookies, the winning tier and timings. Set solveEmbeddedCaptchas: false for page content only; omit it to attempt supported embedded form widgets. Access challenges still need to clear. A delivered page does not by itself prove a form CAPTCHA was solved.

Native API · Custom headers

Prowlarr, Jackett and FlareSolverr clients

curl -X POST http://localhost:8191/v1 \
  -H 'Content-Type: application/json' \
  -d '{"cmd":"request.get","url":"https://nopecha.com/demo/cloudflare"}'

Set your client's FlareSolverr URL to http://trawl:8191 on a shared Docker network, or the TRAWL host address otherwise. /v1 prioritizes page delivery and skips optional embedded widget solving by default.

Prowlarr setup · Jackett setup · Compatibility reference

MCP for AI clients

With MCP_ENABLED=true, connect your client using Streamable HTTP at http://localhost:8191/mcp or the TRAWL host address. Tools: read, scrape, extract, screenshot and inspect, plus the scrape_url compatibility alias.

TRAWL loads known public URLs; it does not provide web search. Existing browser sessions can be reused with sessionId. MCP guide.

Browser sessions

Create an isolated session and pass its ID on subsequent requests:

curl -X POST http://localhost:8191/sessions \
  -H 'Content-Type: application/json' \
  -d '{"id":"research"}'
curl -X POST http://localhost:8191/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://nopecha.com/demo/cloudflare","sessionId":"research","solveEmbeddedCaptchas":false}'
curl -X DELETE http://localhost:8191/sessions/research

Sessions preserve browser storage between requests and keep login state out of the shared clearance cache. Requests within one session are sequential. Sessions expire when idle and do not survive an API restart; open tabs are closed after each request.

Session API, authentication and lifecycle

Challenge and CAPTCHA support

TRAWL detects supported walls and runs the matching browser flow before returning content. Coverage includes JavaScript and proof-of-work challenges, interactive controls, and supported CAPTCHA widget variants.

Some challenges need a particular browser mode, a different egress IP or an external provider. For example, DataDome Device Check uses the opt-in headful pool; built-in hCaptcha support does not include image grids. Passive reCAPTCHA v3 scripts are not treated as blocking widgets.

The optional 2Captcha fallback supports captured Cloudflare Challenge Pages and supported declarative Turnstile/reCAPTCHA v2 widgets without a site profile. Other documented tasks require a site profile. It is disabled without a provider key; paid-service acceptance remains unverified.

Full coverage matrix and external task catalogue · Solver configuration

HTTP/HTTPS forward proxy

Enable MITM_ENABLED=true and point your HTTP-compatible client at port 8192. Normal traffic is forwarded directly; detected challenge responses escalate through TRAWL's browser tiers. HTTPS clients must trust the generated TRAWL CA. Keep the proxy private and persist its CA directory.

Proxy setup · Install the CA · Prowlarr and other clients

Docker images

| Tags | Use case | | -------------------------------- | ------------------------------------ | | :latest, :1.9.0 | Standard release for modern hardware | | :baseline, :1.9.0-baseline | Older x86 CPUs and NAS kernels | | :nightly, :nightly- | Development builds from dev |

All tags use ghcr.io/germondai/trawl. Both release variants are published for Linux AMD64 and ARM64. Standalone containers and source builds.

Configuration

| Setting | Default | Purpose | | ------------------------------------- | ------- | --------------------------------------------------------- | | BROWSER_POOL_SIZE | 1 | Warm browsers available for concurrent work | | SESSION_CACHE_DRIVER | redis | Choose Redis or single-process memory caching | | SCRAPE_MAX_TIMEOUT_MS | 60000 | Default scrape deadline; request timeouts take precedence | | TIER1_TRANSPORT | bun | Set wreq for optional Firefox TLS/HTTP2 impersonation | | BROWSER_SESSION_MAX_ENTRIES | 4 | Maximum named browser sessions per instance | | PROXY_URL / RESIDENTIAL_PROXY_URL | unset | Optional upstream proxies and proxy pools |

All settings · Environment example · Upgrade guide

Documentation and development

• Architecture and tier routing
• Clearance cache and HTTP reuse
• Browser pool and resource management
• Troubleshooting
• Release history
• Contributing and local development

Star History

Contributors

Made with ❤️ by @germondai