Improve Visual Testing Doc

[jennifer-shehane]

Original comment here: cypress-io/cypress#3324 (comment)

• We need to clearly communicate best practices around performing screenshot comparison.
• We need to communicate why the screenshot size is different from the viewport size.
• We need to communicate how screenshots are different when in cypress open versus cypress run
• We need to communicate how scaling is handled and not handled.
• When do you screenshot - how do you make sure everything is loaded How to ensure all react components are loaded before cy.screenshot() cypress#5275

[x-yuri]

• cypress open will run the tests and take screenshots within your current environment. So, your computer's screen size and your browser's size.
• cypress run will run the tests and take screenshots within a headless environment (by . default), in Xvfb, which will be limited to the 1280x720 screen size and the max width of the browser.

This information seems to be obsolete. At least on Linux. cypress open and cypress run by default don't employ xvfb. But cypress run starts headless (new BrowserWindow({show: false})).

One can use DISPLAY= cypress ... to make it start xvfb. Or run xvfb yourself:

$ xvfb-run --server-args='-screen 0 2000x2000x24' npx cypress run

Regardless of using xvfb or not screenshots are limited by the browser dimensions, no matter whether you see it or not (headless). And browser dimensions are limited by videobuffer (xvfb) or physical screen dimensions.

I didn't look into it but I guess that the main difference between run and open is that there's no main window in the former case. That is, run supposedly opens a window (starts a runner) for every spec.

As such I wonder why it's not a good idea to compare open- and run-produced screenshots.

We need to communicate why the screenshot size is different from the viewport size.

Before making a screenshot (fullPage), cypress tries to set zoom level to 100%: AUT width = Math.min(browser width, viewport width). But if viewport width > browser width, it gives up, and zoom level of the resulting screenshot is less than 100%. As such, sizes differ.

[raghugitrepo]

@jennifer-shehane when are we getting above guide for visual testing? I am now at poc stage for my app. Due to async behaviour I am not getting expected images without waits. The documentation and examples for visual regression testing is very limited. Would be fine if you guys provide them soon.

[x-yuri]

Due to async behaviour I am not getting expected images without waits

Does it really have to do with Visual Testing? What exactly are you trying to test?

[piotr-bejamas]

@jennifer-shehane any updates on this? I'm writing some visual tests with Cypress and Percy and I would like to know how to handle with lazy loading/infinite scrolling pages or how to wait for components with some css transitions/animations.

[bahmutov]

For now see this presentation:

• https://slides.com/bahmutov/visual-testing and video https://youtu.be/gUFdU5fQs4o and blog post https://glebbahmutov.com/blog/open-source-visual-testing-of-components/

[olufunbi]

I am experiencing the same issue where screenshots were taken using cypress open and the screenshots taken using cypress run have different sizes. Reading through this thread, I see how that is possible, however, as earlier stated, there needs to be proper documentation about this(Visual testing) in general so they do's and dont's are made crystal clear.

My predicament now is, I have to set different baselines when I want to use cypress run and then delete all baseline and set new ones each time I use cypress open. This can be a pain especially when the project is a large one.

[tfrijsewijk]

I'm experiencing something interesting.. Here are two differences with open (left) vs run (right):

• Scrollbar for <select multiple> is wider
  
• The textarea drag 'n pull corner is bigger
  
• And my favourite one:
  
  "Choose File" vs "Choose file" 🤯

I have a 4K screen, but I'm not enforcing a viewport anywhere. I do this instead:

// cypress/plugins/index.js
module.exports = (on, config) => {
  // `on` is used to hook into various events Cypress emits
  // `config` is the resolved Cypress config
  require('cypress-mochawesome-reporter/plugin')(on);

  addMatchImageSnapshotPlugin(on, config);

  on('before:browser:launch', (browser = {}, launchOptions) => {
    if (browser.family === 'chromium' && browser.name !== 'electron') {
      launchOptions.args.push('--high-dpi-support=1');
      launchOptions.args.push('--force-device-scale-factor=1');
      launchOptions.args.push('--disable-gpu');
    }

    return launchOptions;
  });
};