website-scraper-puppeteer
Plugin for website-scraper which returns html for dynamic websites using puppeteer.Sponsors
Maintenance of this project is made possible by all the contributors and sponsors. If you'd like to sponsor this project and have your avatar or company logo appear below click here. 💖
Requirements
- nodejs version >= 20
- website-scraper version >= 5
Installation
npm install website-scraper website-scraper-puppeteerInstalling Chrome
Puppeteer normally downloads a compatible version of Chrome automatically during installation via a postinstall script. Starting with npm v12, lifecycle scripts (including postinstall) are disabled by default, so this automatic download no longer runs. You may then see an error like Could not find Chrome (ver. ...) when scraping.
If you use npm v12 or newer (or another package manager that blocks install scripts, e.g. recent pnpm), install Chrome manually after installing the packages:
npx puppeteer browsers install chromeAlternatives:
- Re-enable install scripts by adding
puppeteerto theallowScriptsfield in yourpackage.json(or.npmrc), so the browser is downloaded automatically again. - Use a Chrome/Chromium already installed on your system by pointing Puppeteer to it via
launchOptions.executablePath(or thePUPPETEER_EXECUTABLE_PATHenvironment variable).
Usage
import scrape from 'website-scraper';
import PuppeteerPlugin from 'website-scraper-puppeteer';
await scrape({
urls: ['https://www.instagram.com/gopro/'],
directory: '/path/to/save',
plugins: [
new PuppeteerPlugin({
launchOptions: { headless: "new" }, / optional /
gotoOptions: { waitUntil: "networkidle0" }, / optional /
scrollToBottom: { timeout: 10000, viewportN: 10 }, / optional /
exclusiveRendering: true, / optional /
})
]
});
launchOptions - (optional)* - puppeteer launch options, can be found in puppeteer docs
gotoOptions - (optional)* - puppeteer page.goto options, can be found in puppeteer docs
scrollToBottom - (optional)* - in some cases, the page needs to be scrolled down to render its assets (lazyloading). Because some pages can be really endless, the scrolldown process can be interrupted before reaching the bottom when one or both of the bellow limitations are reached:
* timeout - in milliseconds
* viewportN - viewport height multiplier
exclusiveRendering - (optional, default true)* - render pages one at a time and never while other requests are in flight, so cookies rotated by a page (see Cookies) are never presented stale. Set to false for websites which don't rotate cookies: pages are then rendered concurrently, which is considerably faster on large websites.
Cookies
Cookies passed inrequest.headers.Cookie (website-scraper option) are set into the browser's cookie jar for the scraped urls' hosts (including their subdomains), so pages opened in puppeteer send them too — but they are not leaked to other domains.
This also applies when a scraped url redirects to a different host (e.g. example.com → shop.example.net): the redirect target does not receive the configured cookies unless its host is listed in urls.
The jar is the source of truth afterwards: if the website rotates or refreshes cookies (via Set-Cookie, including on redirects, or js executed on the page), the updated cookies are used for all subsequent requests.
To keep rotated cookies consistent, pages are rendered one at a time and never while other requests are in flight, regardless of requestConcurrency (see the exclusiveRendering option to disable this).
await scrape({
urls: ['https://example.com/'],
directory: '/path/to/save',
request: {
headers: { Cookie: 'session=abc123' }
},
plugins: [ new PuppeteerPlugin() ]
});