ScrapFly Scrapers 🕷️
This repository contains educational example scrapers for popular web scraping targets using the ScrapFly web scraping API and Python. Most Scrapers use a simple web scraping stack:
- Python version 3.10+
- Scrapfly's Python SDK for sending HTTP request, bypass blocking and parsing the HTML using the built-in parsel selector.
- asyncio for writing concurrent code using the async/await syntax.
- JMESPath and nested-lookup for JSON parsing when needed.
- loguru for logging.
For AI agents and LLM tools, llms.txt lists every scraper with its functions and sample output, and explains how to run a scraper and how to write a new one.
List of Scrapers
Below is the list of available web scrapers for the supported domains along with their scrape guide, sample datasets, and status. 👇 The status comes from the nightly test run, which also orders the table: scrapers whose tests pass come first, failing and untested ones last.
