Puppeteer tests are flaky on CI

[ylemkimon]

FAIL test/e2e/ClientOptions.test.js
  ● Client console.log › liveReload disabled (ws)
  ● Client console.log › liveReload & hot are disabled (ws)
  ● sockjs client port, no path › browser client › uses correct port and path
FAIL test/e2e/Socket-injection.test.js
  ● ws websocket client injection › testing when hot is enabled and liveReload is disabled › should be injected

    Failed to launch the browser process!
    Inconsistency detected by ld.so: ../elf/dl-tls.c: 481: _dl_allocate_tls_init: Assertion `listp->slotinfo[cnt].gen <= GL(dl_tls_generation)' failed!


    TROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md

      at onClose (node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserRunner.js:193:20)
      at ChildProcess.<anonymous> (node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserRunner.js:184:79)

• https://github.com/webpack/webpack-dev-server/runs/1409566428
• https://github.com/webpack/webpack-dev-server/runs/1406463962
• https://github.com/webpack/webpack-dev-server/runs/1406343832
• https://github.com/webpack/webpack-dev-server/runs/1396967631
• https://github.com/webpack/webpack-dev-server/runs/1108036194

FAIL test/e2e/TransportMode.test.js
  ● transportMode client › custom client › on browser client › logs correctly

    expect(received).toMatchSnapshot()

    Snapshot name: `transportMode client custom client on browser client logs correctly 1`

    - Snapshot  - 5
    + Received  + 1

      Array [
        "Hey.",
    -   "open",
    -   "liveReload",
    -   "[webpack-dev-server] Live Reloading enabled.",
    -   "hash",
    -   "ok",
    +   "Failed to load resource: net::ERR_CONNECTION_REFUSED",
        "close",
        "[webpack-dev-server] Disconnected!",
      ]

      82 |                       }
      83 |                     }
    > 84 |                     expect(res).toMatchSnapshot();
         |                                 ^
      85 |                     done();
      86 |                   });
      87 |                 });

      at test/e2e/TransportMode.test.js:84:33

FAIL test/e2e/TransportMode.test.js (16.054 s)
  ● transportMode client › sockjs › on browser client › logs correctly

    expect(received).toMatchSnapshot()

    Snapshot name: `transportMode client sockjs on browser client logs correctly 1`

    - Snapshot  - 1
    + Received  + 1

      Array [
        "Hey.",
    -   "[webpack-dev-server] Live Reloading enabled.",
    +   "Failed to load resource: net::ERR_CONNECTION_REFUSED",
        "[webpack-dev-server] Disconnected!",
      ]

      82 |                       }
      83 |                     }
    > 84 |                     expect(res).toMatchSnapshot();
         |                                 ^
      85 |                     done();
      86 |                   });
      87 |                 });

      at test/e2e/TransportMode.test.js:84:33

• https://github.com/webpack/webpack-dev-server/runs/1406731042
• https://github.com/webpack/webpack-dev-server/runs/1406126854

FAIL test/e2e/TransportMode.test.js (16.816 s)
  ● transportMode client › custom client › on browser client › logs correctly
  ● transportMode client › ws › on browser client › logs correctly

    Protocol error (Page.navigate): Target closed.

      58 |           runBrowser().then(({ page, browser }) => {
      59 |             const res = [];
    > 60 |             page.goto(`http://localhost:${port}/main`);
         |             ^
      61 |             page.on('console', ({ _text }) => {
      62 |               res.push(_text);
      63 |             });

      at node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:208:63
      at CDPSession.send (node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:207:16)
      at navigate (node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:106:47)
      at FrameManager.navigateFrame (node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:89:13)
      at Frame.goto (node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:405:41)
      at Page.goto (node_modules/puppeteer/lib/cjs/puppeteer/common/Page.js:826:53)
      at test/e2e/TransportMode.test.js:60:13
          at runMicrotasks (<anonymous>)

• https://github.com/webpack/webpack-dev-server/runs/1406370533
• https://github.com/webpack/webpack-dev-server/runs/1406002795

ERROR: Failed to set up Chromium r782078! Set "PUPPETEER_SKIP_DOWNLOAD" env variable to skip download.
Error: getaddrinfo ENOTFOUND storage.googleapis.com
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26) {
  errno: 'ENOTFOUND',
  code: 'ENOTFOUND',
  syscall: 'getaddrinfo',
  hostname: 'storage.googleapis.com'
}

• https://github.com/webpack/webpack-dev-server/runs/1432768906

[alexander-akait]

Any ideas?

[shaodahong]

Maybe we can use docker to run test, better than local puppeteer to run

[hiroppy]

Can we migrate to https://github.com/microsoft/playwright?

[alexander-akait]

Maybe it is not bad idea to try playwright

[ylemkimon]

Maybe we can use docker to run test, better than local puppeteer to run

I've used Selenium in other projects and had a not bad experience. Its error rate is about one in two hundred.

Can we migrate to https://github.com/microsoft/playwright?

I haven't tried playwright, but the low number of issues seems nice.

[hiroppy]

playwright provides chromium, firefox, and webkit and we have some browser issues such as #2466 (comment). yes so it's better to try it.

[alexander-akait]

If somebody want to try it and help to migrate feel free to send a PR, I think it will be a good experience

[snitin315]

@alexander-akait How should we do this, migrate all test at once or one by one in separate PRs then remove puppeteer in last?

[alexander-akait]

I think we can do it in one PR, because API is almost same, we can encapsulate some helpers functions if it will make migration easy

[mahdikhashan]

@snitin315 @alexander-akait hey, I'm an applicant for the gsoc-2024. I started a PR for this issue which has been label relevant to gsoc as well. could you let me know this can be considered as a patch or an early contribution which is required in the proposal?

PR: https://github.com/webpack/webpack-dev-server/pull/5080/files