Puppeteer on Time Friend

How Puppeteer Disguises Itself in Work (Crawler and Anti-Crawler)

Sat, 29 Aug 2020 20:32:05 +0000

To better protect our data and program security.

Today I’ll introduce how to detect if the web program accessing us is a headless browser, and some of their anti-detection methods.

Webdriver Detection

Generally, if in headless browser mode, navigator.webdriver will return true.

Detection Method

Simplest detection method

if (navigator.webdriver) {
 // Operations targeting headless browsers
}

How to Detect After Using defineProperty to Delete webdriver

If the other party uses the following method to delete the webdriver property, there’s still a way to detect it

Node.js Crawler Tips - Download Images or Files with Puppeteer

Tue, 23 Oct 2018 20:32:05 +0000

Some websites’ images don’t allow external links. If we want to download these websites’ images, Puppeteer is a good solution. Today I’ll introduce a crawler tip to download images or files you want through Puppeteer.

How to Download Through Puppeteer?

We need an additional dependency library fs-extra

Code is very simple, main code can be simplified to two or three lines

 const fse = require('fs-extra'); // Required dependency library, you can use native fs if you want

 // Main code
 var viewSource = await page.goto(url);
 await fse.outputFile(`path`, await viewSource.buffer()) // Download to path you want

CentOS7 Environment Puppeteer Startup Failure Solution

Sun, 25 Feb 2018 19:33:33 +0000

Tip: In CentOS6 environment, Puppeteer cannot be started. In CentOS6, many Puppeteer dependency libraries don’t exist. So if you want to run Puppeteer on CentOS6, I suggest giving up, it’s more worthwhile to spend time elsewhere.

Missing Dependencies

Running Puppeteer on a new CentOS7 often fails, many times reporting errors like:

...node_modules/puppeteer/.local-chromium/linux-496140/chrome-linux/chrome: error while loading shared libraries: libpangocairo-1.0.so.0: cannot open shared object file: No such file or directory

This error indicates missing dependencies.

A Headless Chrome Browser: Puppeteer

Thu, 21 Dec 2017 20:32:05 +0000

Puppeteer Introduction

Puppeteer is a Node.js library that provides a set of APIs to control Chrome. Compared to PhantomJS, it can directly call Chrome’s API in your Node.js project. Many automation tests and crawlers can be based on Puppeteer in the future.

Installation

npm install puppeteer

Usage

Screenshot

const puppeteer = require('puppeteer');

(async () => {
 const browser = await puppeteer.launch();
 const page = await browser.newPage();
 await page.goto('https://example.com');
 await page.screenshot({path: 'example.png'});

 await browser.close();
})();

Save Page as PDF

const puppeteer = require('puppeteer');

(async () => {
 const browser = await puppeteer.launch();
 const page = await browser.newPage();
 await page.goto('https://news.ycombinator.com', {waitUntil: 'networkidle2'});
 await page.pdf({path: 'hn.pdf', format: 'A4'});

 await browser.close();
})();

Puppeteer also provides many Chrome APIs https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md