Avoid Puppeteer or Playwright for Web Scraping
I don't suggest to use Puppeteer and Playwright for web scraping.The reasons are very simple: 2021-05-21 05:26:00 Author: incolumitas.com(查看原文) 阅读量:321 收藏

I don't suggest to use Puppeteer and Playwright for web scraping.

The reasons are very simple:

  1. Both libraries use pre-shipped chromium binaries that are not used by real human beings for normal browsing.
  2. It is possible to detect that a browser is automated by those libraries in many different ways.

For example, puppeteer uses a plethora of command line flags that you normally would not use when you start a browser:

  defaultArgs(options: BrowserLaunchArgumentOptions = {}): string[] {
    const chromeArguments = [
      '--disable-background-networking',
      '--enable-features=NetworkService,NetworkServiceInProcess',
      '--disable-background-timer-throttling',
      '--disable-backgrounding-occluded-windows',
      '--disable-breakpad',
      '--disable-client-side-phishing-detection',
      '--disable-component-extensions-with-background-pages',
      '--disable-default-apps',
      '--disable-dev-shm-usage',
      '--disable-extensions',
      '--disable-features=Translate',
      '--disable-hang-monitor',
      '--disable-ipc-flooding-protection',
      '--disable-popup-blocking',
      '--disable-prompt-on-repost',
      '--disable-renderer-backgrounding',
      '--disable-sync',
      '--force-color-profile=srgb',
      '--metrics-recording-only',
      '--no-first-run',
      '--enable-automation',
      '--password-store=basic',
      '--use-mock-keychain',
      // TODO(sadym): remove '--enable-blink-features=IdleDetection'
      // once IdleDetection is turned on by default.
      '--enable-blink-features=IdleDetection',
    ];
    const {
      devtools = false,
      headless = !devtools,
      args = [],
      userDataDir = null,
    } = options;
    if (userDataDir)
      chromeArguments.push(`--user-data-dir=${path.resolve(userDataDir)}`);
    if (devtools) chromeArguments.push('--auto-open-devtools-for-tabs');
    if (headless) {
      chromeArguments.push('--headless', '--hide-scrollbars', '--mute-audio');
    }
    if (args.every((arg) => arg.startsWith('-')))
      chromeArguments.push('about:blank');
    chromeArguments.push(...args);
    return chromeArguments;
  }

But there is more. The DOM and the window object of each page that is created with each call to await browser.newPage() sets up some puppeteer/playwright specific properties that are easy to detect. The easiest to detect is probably navigator.webdriver (even though I am not sure if puppeteer still uses this).

But aren't there npm modules such as puppeteer-extra-plugin-stealth that are specifically developed to hide the fact that puppeteer is used? Yes they exist, but they are very limited in their efficacy. It's just very hard to make puppeteer/playwright unseen.

So what is the solution?

The problem is that every scraping developer is either using puppeteer or playwright. Therefore, all the detection developers have to do is to detect puppeteer or playwright.

My suggestion is: Ditch the browser automation frameworks altogether!

We don't really have to use them anyhow, they spoiled us for too long.

Think about it: What do we really need for browser automation?

All that is required for robust browser automation is a straightforward way to find the coordinates for CSS selectors and a way to obtain the HTML code of the current website.

Obtaining the coordianates for a CSS selector

Put differently, we need a way to fulfill the following function:

f(css_selector) => x and y coordinate

All the clicks, filling out forms, scrolling and other browser automation is done with desktop level browser automation instead of puppeteer's page.click(), page.type() and so on.

We only require the CDP to translate CSS selectors into coordinates.

That can be done with the small CDP library

const CDP = require('chrome-remote-interface');

async function getCoords(css_selector) {
  let client;
  try {
    // connect to endpoint
    client = await CDP();
    // extract domains
    const { Runtime} = client;
    // enable events then start!
    await Promise.all([Runtime.enable()]);

    // get clientRect of links
    const result = await Runtime.evaluate({
      expression: `let node = document.querySelector('${css_selector}'); JSON.stringify(node.getClientRects());`
    });

    // get screen positioning
    const screenPos = await Runtime.evaluate({
      expression: "JSON.stringify({top: window.screen.height - window.innerHeight})"
    });

    let baseYOffset = JSON.parse(screenPos.result.value).top;
    let coords = result.result.value[0][0];

    return {
      x: coords.x + getRandomInt(5, coords.width-1),
      y: baseYOffset + coords.y + getRandomInt(5, coords.height-1),
    });

  } catch (err) {
    console.error(err);
  } finally {
    if (client) {
      await client.close();
    }
  }
}

Obtaining the HTML of the current page

Grabbing the HTML code of the current page can also be implemented in a straightforward fashion with the CDP:

const CDP = require('chrome-remote-interface');

async function getPageSource() {
  let client;
  try {
    // connect to endpoint
    client = await CDP();
    // extract domains
    const { Page, Runtime, DOM } = client;
    // enable events then start!
    await Promise.all([Page.enable(), Runtime.enable(), DOM.enable()]);

    // get the page source
    const rootNode = await DOM.getDocument({ depth: -1 });
    const pageSource = await DOM.getOuterHTML({
      nodeId: rootNode.root.nodeId
    });
    return pageSource.outerHTML;
  } catch (err) {
      console.error(err);
  } finally {
    if (client) {
      await client.close();
    }
  }
}

文章来源: https://incolumitas.com/2021/05/20/avoid-puppeteer-and-playwright-for-scraping/
如有侵权请联系:admin#unsafe.sh