Sometimes, we want to scroll down until you can’t anymore with Puppeteer.
In this article, we’ll look at how to scroll down until you can’t anymore with Puppeteer.
How to scroll down until you can’t anymore with Puppeteer?
To scroll down until you can’t anymore with Puppeteer, we can scroll periodically down the page until the height scrolled is bigger than or equal to the scroll height of the scroll container.
For instance, we write
const puppeteer = require('puppeteer');
const autoScroll = async (page) => {
await page.evaluate(async () => {
await new Promise((resolve, reject) => {
let totalHeight = 0;
const distance = 100;
const timer = setInterval(() => {
const scrollHeight = document.body.scrollHeight;
window.scrollBy(0, distance);
totalHeight += distance;
if (totalHeight >= scrollHeight) {
clearInterval(timer);
resolve();
}
}, 100);
});
});
}
const openPageAndScroll = async () => {
const browser = await puppeteer.launch({
headless: false
});
const page = await browser.newPage();
await page.goto('https://www.yoursite.com');
await page.setViewport({
width: 1200,
height: 800
});
await autoScroll(page);
//...
await browser.close();
}
openPageAndScroll()
to define the autoScroll function.
In it, we call setInterval with a callback that calls window.scrollBy to scroll by distance pixels down.
We then add distance to totalHeight.
And then if totalHeight >= scrollHeight, we call clearInterval to clear the timer and call resolve to resolve the promise.
And we scroll every 100ms.
Next, we define the openPageAndScroll function that calls autoScroll and use await to wait for the promise to resolve before running the next line.
Conclusion
To scroll down until you can’t anymore with Puppeteer, we can scroll periodically down the page until the height scrolled is bigger than or equal to the scroll height of the scroll container.
