Continuous Integration

The tutorial you are about to read will show you how you can run Cypress tests in Continuous Integration, we will show you how to configure Cypress for various CI Providers. This tutorial will also show you how you can record tests to the Cypress Dashboard. Finally, we will show you how to run tests in parallel on a CI.

Setting up CI

Basics

Running Cypress in Continuous Integration is very similar to running Cypress locally in your terminal. In general, you will only need to do two things:

1. Install Cypress
   npm install cypress --save-dev
2. Run Cypress
   Run Cypress

Depending on the CI provider that you are using, you may have to create a config file. You may also have to refer to the documentation of your provider, in order to know where to add the commands to install and run Cypress.

Boot your server

Challenges

Typically, you will have to boot a local server before running Cypress. Whenever you boot your web server, it will run as long as there is a long running process that will never exit. This implies that you have to run the server in the background, else your CI provider will not be able to move onto the next command.

Running your server in the background implies that your CI provider will continue to execute the next command after it executes the signal to start your server.

A poor way to implement this is to run the following command:

'''npm start & cypress run // You should not do this'''

The problem with this is that if the server takes time to boot, you don?t have a guarantee that when the next command run (cypress run), your web server will still be up and available. Hence, your Cypress test may start and try to visit your local server before it is ready to be visited.

Solutions

There are some workarounds to this challenge. Rather than introducing arbitrary waits (such as sleep 20), you can use a better option.

A. wait-on module

When you use the wait-on module, you can block the cypress run command from being executed until your server has booted.

'''npm start & wait-on https://localhost:8080'''

'''cypress run'''

B. start-server-and-test module

In the case where the server takes long to start, it is recommended that you try the start-server-and-test module.

npm install --save-dev start-server-and-test

Once you install the module, you then need to pass the command to boot your server, the url that your server is hosted on and your Cypress test command.

'''{
  "scripts": {
    "start": "my-server -p 3030",
    "cy:run": "cypress run",
    "test": "start-server-and-test start https://localhost:3030 cy:run"
  }
}'''

In the example shown above, the cy:run command is only executed when the URL https://localhost:3030 responds with an HTTP status code of 200. The server will also shut down once the tests complete.

Gotchas

Whenever you are working with webpack-dev-server which does not respond to HEAD requests, you should use an explicit GET method to ping the server like this:

'''{
  "scripts": {
    "test": "start-server-and-test start http-get://localhost:3030 cy:run"
  }
}'''

When you are working with local https in webpack, you should set an environment variable to allow local certificate:

'''{
  "scripts": {
    "start": "my-server -p 3030 --https",
    "cy:run": "cypress run",
    "cy:ci": "START_SERVER_AND_TEST_INSECURE=1 start-server-and-test start https-get://localhost:3030 cy:run"
  }
}'''

Record tests

You can use Cypress to record your tests and make the results of the tests available in the Cypress Dashboard.

Recording tests enable you to:

• See the number of passing, pending and failed tests.
• Get the entire stack trace of your failed tests.
• View screenshots taken when tests fail and when using cy.screenshot().
• Watch a video of your entire test run or a clip at the point of test failure.
• See which machines ran each test when parallelized.

In order to record tests, you will need to:

1. Set up your project to record
2. Pass the --record flag to cypress run within your CI.

'''cypress run --record --key=abc123'''

Run tests in parallel

Cypress is able to run tests in parallel across multiple machines. You should refer to the documentation of your CI provider on how to set up multiple machines to run in your CI environment.

Once multiple machines are available in your CI environment, you can then pass --parallel flag, so as to have your tests run in parallel.

'''cypress run --record --key=abc123 -parallel'''

Examples

Cypress can run on all CI providers, the Cypress team has provided some example projects and configuration for some CI providers in order to help you get started.

Travis

Example .travis.yml config file:

'''language: node_js
node_js:
  - 10
addons:
  apt:
    packages:
      # Ubuntu 16+ does not install this dependency by default, 
so we need to install it ourselves
      - libgconf-2-4
cache:
  # Caches $HOME/.npm when npm ci is default script command
  # will cache node_modules in all other cases
  npm: true
  directories:
    # we will also need to cache folder with Cypress binary
    - ~/.cache
install:
  - npm ci
script:
  - $(npm bin)/cypress run --record'''

Caching folders with npm modules will save a lot of time after the first build.

CircleCI

The Cypress CircleCI Orb is a piece of configuration file that is set in your circle.yml file to correctly install, cache and run Cypress with very little effort.

A typical CircleCI Project can be as shown below:

version: 2.1

'''orbs:
  # "cypress-io/[email protected]" installs the latest published
  # version "1.x.y" of the orb. We recommend you then use
  # the strict explicit version "cypress-io/[email protected]"
  # to lock the version and prevent unexpected CI changes
  cypress: cypress-io/[email protected]
workflows:
  build:
    jobs:
      - cypress/run # "run" job comes from "cypress" orb'''

However, a more complex project that requires you to install dependencies, build an application and run tests across 4 CI machines in parallel may have:

version: 2.1

'''orbs:
  cypress: cypress-io/[email protected]
workflows:
  build:
    jobs:
      - cypress/install:
          build: 'npm run build'  # run a custom app build step
      - cypress/run:
          requires:
            - cypress/install
          record: true        # records the results on Cypress Dashboard
          parallel: true      # splits all the specs across machines
          parallelism: 4      # use 4 CircleCI machines to finish quickly
          group: 'all tests'  # name this group "all tests" on the dashboard
          start: 'npm start'  # start server before running tests'''

In most cases you will be using run and install job definitions that Cypress provides inside the orb. Using the orb will bring simplicity and static checks of parameters to CircleCI configuration.

Example circle.yml v2 config file.

version: 2

'''jobs:
  build:
    docker:
      - image: cypress/base:8
        environment:
          ## this will enable colors in the output
          TERM: xterm
    working_directory: ~/app
    steps:
      - checkout
      - restore_cache:
          keys:
            ```- v1-deps-{{ .Branch }}-{{ checksum "package.json" }}```
            - v1-deps-{{ .Branch }}
            - v1-deps
      - run:
          name: Install Dependencies
          command: npm ci
      - save_cache:
          key: ```v1-deps-{{ .Branch }}-{{ checksum "package.json" }}```
          # will cache the NPM modules and the folder using the Cypress binary
          paths:
            - ~/.npm
            - ~/.cache
      - run: $(npm bin)/cypress run --record --key <record_key>'''

Example circle.yml v2 config file using yarn

version: 2

'''jobs:
  build:
    docker:
      - image: cypress/base:8
        environment:
          ## this will enable colors in the output
          TERM: xterm
    working_directory: ~/app
    steps:
      - checkout
      - restore_cache:
          keys:
            ```-v1-deps-{{ .Branch }}-{{ checksum "package.json" }}```
            - v1-deps-{{ .Branch }}
            - v1-deps
      - run:
          name: Install Dependencies
          command: yarn install --frozen-lockfile
      - save_cache:
          key: ```v1-deps-{{ .Branch }}-{{ checksum "package.json" }}```
          paths:
            - ~/.cache  ## this will cache both yarn and Cypress!
      - run: $(yarn bin)/cypress run --record --key <record_key>'''

AWS Amplify

Example amplify.yml

version: 0.1

'''frontend:
  phases:
    preBuild:
      commands:
        - npm install
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: app
    files:
      - "**/*"
  cache:
    paths:
      - node_modules/**/*
test:
  artifacts:
    baseDirectory: cypress
    configFilePath: "**/mochawesome.json"
    files:
      - "**/*.png"
      - "**/*.mp4"
  phases:
    preTest:
      commands:
        - npm install
        - npm install wait-on
        - npm install mocha mochawesome mochawesome-merge mochawesome-report-generator
        - "npm start & npx wait-on https://127.0.0.1:8080"
    test:
      commands:
        - 'npx cypress run --reporter mochawesome --reporter-options "reportDir=cypress/report/mochawesome-report,overwrite=false,html=false,json=true,timestamp=mmddyyyy_HHMMss"'
    postTest:
      commands:
        - npx mochawesome-merge --reportDir cypress/report/mochawesome-report > cypress/report/mochawesome.json'''

Example amplify.yml v2 with --record for Cypress Dashboard.

You will need to add CYPRESS_RECORD_KEY Environment Variable in Amplify Console.

version: 0.1

'''frontend:
  phases:
    preBuild:
      commands:
        - npm install
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: app
    files:
      - "**/*"
  cache:
    paths:
      - node_modules/**/*
test:
  artifacts:
    baseDirectory: cypress
    configFilePath: "**/mochawesome.json"
    files:
      - "**/*.png"
      - "**/*.mp4"
  phases:
    preTest:
      commands:
        - npm install
        - npm install wait-on
        - npm install mocha mochawesome mochawesome-merge mochawesome-report-generator
        - "npm start & npx wait-on https://127.0.0.1:8080"
    test:
      commands:
        - 'npx cypress run --record --reporter mochawesome --reporter-options "reportDir=cypress/report/mochawesome-report,overwrite=false,html=false,json=true,timestamp=mmddyyyy_HHMMss"'
    postTest:
      commands:
        - npx mochawesome-merge --reportDir cypress/report/mochawesome-report > cypress/report/mochawesome.json'''

Example amplify.yml for Create React App and Amplify Console

version: 0.1

'''backend:
  phases:
    build:
      commands:
        - "# Execute the Amplify CLI with the helper script"
        - amplifyPush --simple
frontend:
  phases:
    preBuild:
      commands:
        - yarn install
    build:
      commands:
        - yarn run build
  artifacts:
    baseDirectory: build
    files:
      - "**/*"
  cache:
    paths:
      - node_modules/**/*
test:
  artifacts:
    baseDirectory: cypress
    configFilePath: "**/mochawesome.json"
    files:
      - "**/*.png"
      - "**/*.mp4"
  phases:
    preTest:
      commands:
        - npm install
        - npm install wait-on
        - npm install mocha mochawesome mochawesome-merge mochawesome-report-generator
        - "npm start & npx wait-on https://localhost:3000"
    test:
      commands:
        - 'npx cypress run --reporter mochawesome --reporter-options "reportDir=cypress/report/mochawesome-report,overwrite=false,html=false,json=true,timestamp=mmddyyyy_HHMMss"'
    postTest:
      commands:
        - npx mochawesome-merge --reportDir cypress/report/mochawesome-report > cypress/report/mochawesome.json'''

Example amplify.yml for Create React App and Amplify Console with –record for Cypress Dashboard.

You should add CYPRESS_RECORD_KEY Environment Variable in Amplify Console.

version: 0.1

'''backend:
  phases:
    build:
      commands:
        - "# Execute the Amplify CLI with the helper script"
        - amplifyPush --simple
frontend:
  phases:
    preBuild:
      commands:
        - yarn install
    build:
      commands:
        - yarn run build
  artifacts:
    baseDirectory: build
    files:
      - "**/*"
  cache:
    paths:
      - node_modules/**/*
test:
  artifacts:
    baseDirectory: cypress
    configFilePath: "**/mochawesome.json"
    files:
      - "**/*.png"
      - "**/*.mp4"
  phases:
    preTest:
      commands:
        - npm install
        - npm install wait-on
        - npm install mocha mochawesome mochawesome-merge mochawesome-report-generator
        - "npm start & npx wait-on https://localhost:3000"
    test:
      commands:
        - 'npx cypress run --record --reporter mochawesome --reporter-options "reportDir=cypress/report/mochawesome-report,overwrite=false,html=false,json=true,timestamp=mmddyyyy_HHMMss"'
    postTest:
      commands:
        - npx mochawesome-merge --reportDir cypress/report/mochawesome-report > cypress/report/mochawesome.json'''

Docker

The Cypress team has created an official cypress/base container that has all the required dependencies installed. You can add Cypress and go! A typical Dockerfile will look like this:

'''FROM cypress/base'''

'''RUN npm install'''

'''RUN $(npm bin)/cypress run'''

Any attempt to mount a project directory with an existing node_modules into a cypress/base docker image won’t work. Rather, you will need to build a docker container for the cypress version of your project.

Advanced setup

Dependencies

In the case where you are not using one of the above CI providers then make sure your system has these dependencies installed.

Linux

Ubuntu/Debian

'''apt-get install libgtk2.0-0 libgtk-3-0 libnotify-dev libgconf-2-4 libnss3 libxss1
libasound2 libxtst6 xauth xvfb'''

CentOS

'''yum install -y xorg-x11-server-Xvfb gtk2-devel gtk3-devel libnotify-devel GConf2
nss libXScrnSaver alsa-lib'''

Caching

As of Cypress version 3.0, Cypress will download its binary to the global system cache - on linux that is ~/.cache/Cypress. By ensuring this cache persists across builds, you will be able to save minutes off install time by preventing a large binary download.

We recommend users:

• Cache the ~/.cache folder after they run npm install, yarn, npm ci or equivalents as demonstrated in the configs below.
• Do not cache the node_modules folder across builds. This will bypass more intelligent caching packaged with npm or yarn, and can cause issues with Cypress not downloading the Cypress binary when for npm install.
• In the case where you are using npm install in your build process, you should consider switching to npm ci and caching the ~/.npm directory for a faster and more reliable build.
• If you are using yarn, caching ~/.cache will include both the yarn and Cypress caches. You should consider using yarn install --frozen-lockfile as an npm ci equivalent.

If you want to override the binary location for some reason, you can use CYPRESS_CACHE_FOLDER environment variable.

Environment variables

You can set various environment variables in order to modify how Cypress runs.

Configuration Values

You can set any configuration value as an environment variable. This overrides values in your configuration file (cypress.json by default).

The typical use cases would be modifying things like:

• CYPRESS_BASE_URL
• CYPRESS_VIDEO_COMPRESSION
• CYPRESS_REPORTER
• CYPRESS_INSTALL_BINARY

Record Key

In the case where you are recording your runs on a public project, you should protect your Record key.

Rather than hard coding it into your run command like this:

'''cypress run --record --key abc-key-123'''

You can set the record key using the environment variable, CYPRESS_RECORD_KEY, and Cypress will automatically use that value. Now you can omit the --key flag when recording.

'''cypress run --record'''

Typically you would set this inside of your CI provider.

Git information

Cypress will use the @cypress/commit-info package to extract git information to associate with the run (e.g. branch, commit message, author).

It will assume that there is a .git folder and then use Git commands to get each property, like git show -s --pretty=%B to get commit message, see src/git-api.js.

In some environment setups (e.g. docker/docker-compose), in the case where the .git directory is not available or mounted, you can pass all the git related information under custom environment variables.

• Branch: COMMIT_INFO_BRANCH
• Message: COMMIT_INFO_MESSAGE
• Author email: COMMIT_INFO_EMAIL
• Author: COMMIT_INFO_AUTHOR
• SHA: COMMIT_INFO_SHA
• Remote: COMMIT_INFO_REMOTE

If the commit information is missing in the Dashboard run, then GitHub Integration or other tasks may not work correctly. In order to see the relevant Cypress debug logs, you should set the environment variable DEBUG on your CI machine and then inspect the terminal output to see why the commit information is unavailable.

'''DEBUG=commit-info,cypress:server:record'''

Custom Environment Variables

You can equally set custom environment variables for use in your tests. These will enable your code to reference dynamic values.

'''export "EXTERNAL_API_SERVER=https://corp.acme.co"'''

And then in your tests:

'''cy.request({
  method: 'POST',
  url: Cypress.env('EXTERNAL_API_SERVER') + '/users/1',
  body: {
    foo: 'bar',
    baz: 'quux'
  }
})'''

Module API

Most times it can be less complex to programmatically control and boot your servers using a Node script.

If you are using Cypress’s Module API, then you can write a script that boots and then shuts down the server later. As a bonus, you can also work with the results and do other things.

'''// scripts/run-cypress-tests.js

const cypress = require('cypress')
const server = require('./lib/my-server')

// start your server
return server.start()
.then(() => {
  // kick off a cypress run
  return cypress.run()
  .then((results) => {
    // stop your server when it's complete
    return server.stop()
  })
})'''

'''node scripts/run-cypress-tests.js'''

Common problems and solutions

Missing binary

Whenever npm or yarn install the cypress package, a postinstall hook will be executed, this hook downloads the platform-specific Cypress binary. If the hook is skipped for any reason, then the Cypress binary will be missing (unless it was already cached).

In order to better diagnose the error, you should add commands to get information about the Cypress cache to your CI setup. This prints where the binary is located and what versions are already present.

'''npx cypress cache path'''

'''npx cypress cache list'''

If you don’t find the required binary version in the cache, you can try following step.

1. Clean your CI’s cache by using your CI’s settings to force a clean npm install on the next build.
2. You can run the binary install yourself by adding the command npx cypress install to your CI script. If there is a binary that is already present, it should finish quickly.

In Docker

In the case where you are running long runs on Docker, you will need to set the ipc to host mode. This issue describes exactly what you have to do.

The default size of the /dev/shm shared memory space for a docker container is 64MB. This will typically not be enough to run Chrome and can cause the browser to crash. To fix this you can pass the --disable-dev-shm-usage flag to Chrome with the following workaround:

'''module.exports = (on, config) => {
  on('before:browser:launch', (browser = {}, launchOptions) => {
    if (browser.family === 'chromium' && browser.name !== 'electron') {
      launchOptions.args.push('--disable-dev-shm-usage')
    }

    return launchOptions
  })
}'''

Xvfb

Whenever you are running on Linux, Cypress will need an X11 server; otherwise it will spawn its own X11 server during the test run. Whenever you are running several Cypress instances in parallel, the spawning of multiple X11 servers at once could cause problems for some of them. In such cases, you can start a single X11 server separately and then pass the server’s address to each Cypress instance using DISPLAY variable.

First, you should spawn the X11 server in the background at some port, for example :99. If you have already installed xvfb on Linux or you are using one of Cypress’s Docker images from cypress-docker-images, the tools below should be available to you.

'''Xvfb :99 & '''

Secondly, you need to set the X11 address in an environment variable

'''export DISPLAY=:99 '''

Then start Cypress as usual:

'''npx cypress run '''

Once the tests across all Cypress instances have finished, you should kill the Xvfb background process using pkill

'''pkill Xvfb '''

Colors

If you want to disable colors, pass the NO_COLOR environment variable to disable colors.

This may be necessary if ASCII characters or colors are not formatted properly in your CI.

'''NO_COLOR=1 cypress run'''