Setting up integration tests in GitLab CI

As the lead developer for the Website Template Project, maintaining quality is a top priority for me. We recently came to a point in our growth at which our regular manual testing was getting costly. Our quality efforts needed to scale. It was time to automate our website tests.

In this article, I’ll review a basic setup for automated integration tests. The process is fairly straightforward, but there are a couple of gotchas that this article might help you avoid. To make it even easier, here’s a little repo with a simple example that we’ll use throughout the discussion.

Docker setup
Test server docker setup
Test runner docker setup
Docker Compose setup
Running tests locally
GitLab CI setup
Wrapping up


The example consists of a GatsbyJS server to test and a simple integration test to run against it.

We want to ensure the tests are scalable and keep the environment consistent no matter where we run the tests. We’ll use Docker to spin up both the test server and test runner, and manage the communication between them.

We just need to set that up, then get it running in GitLab CI.

Here are the files directly relevant to the Docker and CI setups:


It’s a good idea to have some basic knowledge of the following.

  • Cucumber. We’ll be using cucumber-js with selenium for our integration tests. What’s most important for this discussion is that we’re running integration tests that require talking to a server. Everything we’re doing could easily be adapted to test an API, for example.
  • Docker. We’re going to run everything in little controlled environments. For this purpose, you can liken Docker containers to virtual machines.
  • Docker Compose. Makes it easier to work with Docker containers and manage the communication between them.
  • GatsbyJS. A frontend framework that we’ll use to kick off a server. You could use Flask or another preferred way of serving web pages.
  • GitLab CI/CD. Use a simple yaml file to test, deploy, build, and perform other tasks that you’d rather leave to machines.


  1. Fork this repo or copy it into a blank GitLab project.
  2. Install Docker.
  3. Install Docker Compose.

If you want to run the server or tests outside of docker, you’ll need to take a few additional steps. See the README if you’d like to do that.

The README also contains a brief “Troubleshooting” section that can help you overcome common hurdles.

Docker setup

Test server docker setup

First, let’s look at the test server’s Dockerfile:

Let’s go over that one piece at a time.

We use npm to install our server’s node modules and set up Gatsby’s CLI. So we base our Docker image off of the standard node image. If your server uses Flask or Django, you could use a Python image instead.

This section just copies everything in hello-world/ into the image, then sets /test-server/ as the directory where we’ll run our commands.

Once we have our server code in the image, we need to install node modules and build the public-facing files. If you’re using a Python-based server like Flask or Django, this phase would involve a pip install -r, instead. Similarly for other servers.

This is how to spin up a production server with GatsbyJS. Note the --host part. This makes the server accessible to other devices on the network. It’s crucial for our setup because the test runner needs to talk to our server. Other servers have similar flags or settings.

Test runner docker setup

Next is the test runner Dockerfile.

Again, let’s take it step-by-step.

We’re using node as our base image again because the integration tests are written using Javascript. If your tests are written in Ruby, for example, you’d choose a Ruby image.

This is a standard Debian-style install command. Note that I chose Firefox ESR, the Extended Support Release. You can choose whatever version you want your code to support. If you want to test with a different browser, like Chrome, you can install that, instead.

This section copies the integration tests into the image, sets the entry directory to the code’s location, and installs the node modules. As usual, adjust the installation command for your test code. If your tests are written in Ruby, you might use a bundle install.

In order to run integration tests against a browser using Selenium, you’ll need the a Web Driver corresponding to that browser. The code above downloads the web driver for Firefox, decompresses it in the same directory as the test code, and puts adds directory to the PATH. There are different web drivers corresponding to different browsers, so choose the one for the browser(s) you want to test with.

This line sets an environment variable that we use to control whether the browser runs in headless mode. See this section of the cucumber setup code. I always run in headless mode through Docker, but sometimes I like to watch the tests click through the site if I’m running the tests directly on my local machine.

The location and name of the test server changes depending on the context. In the Docker context, we direct our tests towards http://test-server:9000. Outside of the little Docker park, we use http://localhost:9000 if running the server locally, or something like to target a remote server. [Note that the port may vary depending on your server. 9000 is the default port for Gatsby’s production server.] We’ll use Docker Compose to set the TEST_SERVER variable for the Docker context. If running the tests on your local machine, you can use: TEST_SERVER=http://localhost:9000 npm run test. See the README for details.

Does what it says. The command is set to run Cucumber here.

Docker Compose setup

Here are the contents of docker-compose.yaml:

That sets up a couple of services, test-server and test-runner, which can interact via a default network.

For test-server, we tell Docker Compose where to find the Dockerfile and code, and that we want to use port 9000 (Gatsby’s default production server port). We also define a healthcheck so Docker Compose understands what it means for the server to be healthy.

For test-runner, we again start by telling Docker Compose where to find the Dockerfile and code. We then set up the TEST_SERVER environment variable, as discussed in the previous section. Finally, we state that the test runner depends on the server being healthy.

Heads up: I still had to set a timeout for Cucumber tests to pass on slower hosts like the cute little GitLab runners. So there’s some room to optimize the healtheck.

Running tests locally

So far, we’ve handled the Docker and Docker Compose setup. Execute the following from within the project dir:

That kicks off docker containers for the test server and test runner, including the specifications we gave in docker-compose.yaml. The command will exit when test-runner finishes and with the same exit code. So we’ll know if the tests fail. You can check the exit code with echo $? right after running the tests, or do something like:

See the README if you’d like to run the tests outside of the Docker context. If you run into trouble, the README has a little trouble-shooting section.

GitLab CI setup

GitLab CI will automatically pick up instructions in .gitlab-ci.yaml:

This just sets up an integration test step. It’s easy to incorporate this step into a CI that includes other steps.

First, we set the Docker Compose image in our .gitlab-ci.yaml. Do not use latest. I tried and found myself with an older, incompatible version. You can set the version to whatever works for you locally. Check your local version with:

Next, we need to equip the test stage with Docker-in-Docker (dind). We do so by setting the service and related Docker variables as shown. See GitLab’s documentation for more info.

Finally, the script executed is the same one we used to run the tests locally:

Since that command exits with the same exit code as test-runner, the CI step will fail if the tests fail, which is the desired behavior.

Wrapping up

That’s it! Now you should be able to push changes to the project and watch the tests run.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store