Containerized Automated Tests in Azure DevOps

 

Summary: Very good article on end to end testing and using azure dev ops.

In this article, we will talk about how one can easily containerize and run automated test cases as part of CI — CD using Azure DevOps.

Let’s take a quick look at what is meant by containerization in the first place.

Containerization involves bundling an application together with all of its related configuration files, libraries and dependencies required for it to run in an efficient and bug-free way across different computing environments.

If we try to break this down in Test Automation language, containerization will involve bundling the test scripts, framework dependencies, setting files, etc., and then running them against the Dev, Test, Acceptance and sometimes Production environments, in a complete isolated manner.

So, why should we containerize our tests?

Well, the answer is pretty much in the paragraph above. Setting up of automated tests can sometimes be a little complex considering the amount of work that needs to be done in maintaining the browsers and WebDriver versions. In addition, one would need Java or Python or Node.js to be installed, keeping a close tab on the compatible versions!

To give an example, let’s say the latest version of your automation framework is only compatible with Java 11. But the Virtual Machines or Azure agents of your organization is on Java 8. Hmmm! Quite a tight spot you are in, no?

With containerization, all these issues are eliminated. A containerized test runs exactly the same way in any machine, CI or local, irrespective of the underlying OS and other dependencies. Since these tests run on isolated containers, one has an opportunity to scale up the testing to infinity.

This changes the fundamental question from why should we containerize our tests to why should we not containerize our tests?

Up next, we will see how we can achieve this in Azure DevOps. The flow that we will discuss is as below :

You can find the sample project and files referred to in this article, in the following GitHub link : https://github.com/ghoshasish99/CITS

Step 1 : Version control your test automation code. You can choose GIT or Azure Repos depending upon your organization’s requirements. Ensure that the Dockerfile(to build the image) and docker-compose.yml(to create and run test containers) are present in your repository branch.

We will talk about these 2 files shortly.

Step 2: Move over to your Azure DevOps and configure it to be able to build a docker image and push it to a Container Registry. For this you will need to have the following :

a. A Container Registry — this can be ACR (Azure Container Registry), DockerHub Container Registry, Google Container Registry or any other of your choice. For this article I will go for DockerHub Container Registry.

b. A service connection — you can set this up using the following steps.

Go to your Azure Project → Navigate to Project Settings → Select ‘Service connections’ → Click on ‘New service connection’ → From ‘New service connection’, select Docker Registry → In ‘New Docker Registry service connection’ overlay, enter the fields indicated below → Click on ‘Verify and save’.

That’s it! You have established a connection gateway between ADO and Docker Registry.

Step 3: Create an Azure Build Pipeline which will build your automated test docker image and push it to the Docker Registry. For this we will use the following YAML file.

trigger:
- master
resources:
- repo: self
variables:
  tag: '$(Build.BuildId)'
stages:
- stage: Build
  displayName: Build and Push image
  jobs:
   - job: Build
     displayName: Build and Push
     pool:
        vmImage: 'ubuntu-latest'
     steps:
      - task: Docker@2
        inputs:
           containerRegistry: 'Dockerhub'
           repository: 'ghoshasish99/cits-test'
           command: 'buildAndPush'
           Dockerfile: '**/Dockerfile'      - task: CopyFiles@2
        inputs:
           Contents: 'docker-compose.yml'
           TargetFolder: '$(Build.ArtifactStagingDirectory)'      - task: PublishBuildArtifacts@1
        inputs:
           PathtoPublish: '$(Build.ArtifactStagingDirectory)'
           ArtifactName: 'drop'
           publishLocation: 'Container'

Now lets break this YAML file down for a clearer understanding:

- task: Docker@2
        inputs:
           containerRegistry: 'Dockerhub'
           repository: 'ghoshasish99/cits-test'
           command: 'buildAndPush'
           Dockerfile: '**/Dockerfile'

This Docker task asks ADO to use the service connection created earlier (I had named it DockerHub — refer Step 2b above) and push the image to the specified Repository, after ADO has built it from the Dockerfile. It is important to note that the image is tagged with ‘$(Build.BuildId)’ so that its always unique for a particular repository.

I have used CITS as the Test automation tool for this article. My Dockerfile looks like this :

FROM java:openjdk-8-jre
LABEL maintainer="Ashish Ghosh"
ENV PROJ=""
ENV RELEASE=""
ENV TESTSET=""
ENV GRIDURL=""
RUN mkdir /workspace
WORKDIR /workspace
COPY    . .
RUN chmod -R 755 ./
CMD ./Run.command -run -project_location ${PROJ} -release ${RELEASE} -testset ${TESTSET} -setEnv "run.RemoteGridURL=${GRIDURL}"

This uses a base image of openjdk-8-jre as the version of CITS that I am working on, supports only Java8. I have used some environment variables in order to keep my docker image dynamic and hence scalable. So when the image will be built, all I need to do is pass the execution details as environment variables and my docker container is good to run!

For a traditional cucumber-maven project a Dockerfile will typically look like this, with some variations.

FROM    maven:3.6.0-jdk-8
RUN     mkdir /functional-test
WORKDIR /functional-test
COPY    . .
CMD mvn clean test -Dcucumber.options="--tags '@Regression and @Smoke'" -DexecutionPlatform="GRID"

To know details on the best practices of Dockerfile refer to this link : https://docs.docker.com/develop/develop-images/dockerfile_best-practices/

Going back to our YAML file, the next 2 tasks are to publish the Build Artifact, in this case docker-compose.yml file, so that it can later be used by Azure Release Pipelines to pull and run the tests in containers.

- task: CopyFiles@2
        inputs:
           Contents: 'docker-compose.yml'
           TargetFolder: '$(Build.ArtifactStagingDirectory)'- task: PublishBuildArtifacts@1
        inputs:
           PathtoPublish: '$(Build.ArtifactStagingDirectory)'
           ArtifactName: 'drop'
           publishLocation: 'Container'

At the end of this, an Artifact location called “drop” will be created which will contain, the docker-compose.yml file. So the final outcome of the build pipeline should yield the following :

Step 4: Create a Release Pipeline

a. Choose the Build which we created above as the source of Artifacts

b. Create 3 tasks -
i. Run Dockerized Test : This is to pull the images (test automation image, Selenium grid images) and spin up the containers to run the tests.
ii. Publish Test Results : This is to publish the Test Results to Azure DevOps.
iii. Clean up : This is to bring down and remove the containers.

Lets focus on the first task — “Run Dockerized Test”. Its a Command Line task which simply uses the build artifact (drop) docker-compose.yml to pull all the necessary images and starts the test execution.

The “Script” field in the above task executes a docker-compose run with some parameters. In order to understand the script, lets first have a close look at the docker-compose.yml file.

version: "3"
services:
  selenium-hub:
    image: selenium/hub
    container_name: selenium-hub
    ports:
      - "4444:4444"
  chrome:
    image: selenium/node-chrome
    depends_on:
      - selenium-hub
    environment:
      - HUB_HOST=selenium-hub
      - HUB_PORT=4444
  automation:
    image: ghoshasish99/cits-test:${TAG}
    container_name: automation-test
    depends_on:
      - chrome 
    environment:
      - PROJ=Projects/${PROJECT_NAME}
      - RELEASE=${RELEASE_NAME}
      - TESTSET=${TESTSET_NAME}
      - GRIDURL=http://selenium-hub:4444/wd/hub     
    volumes:
      - ./Results/:/workspace/Projects/${PROJECT_NAME}/Results/TestExecution/${RELEASE_NAME}/${TESTSET_NAME}

There are 3 services that are invoked in this docker-compose. The first 2 are to get selenium grid up and running for UI tests. This makes the entire execution atomic and scalable.

The 3rd service is the execution of the actual tests. It pulls the test automation image from the container registry and spins up a container. It is important to note the usage of the following variables which makes the execution dynamic.

- ${TAG} : This is the image tag that needs to be pulled. If you remember this is the $(Build.BuildId) which was used to tag the image before pushing to the registry.- ${PROJECT_NAME} , ${RELEASE_NAME} , ${TESTSET_NAME} are certain variables defined specifically to pass as inputs to the CITS execution engine.

So the final command used in the task to run docker-compose is as follows :

TAG=$(Release.Artifacts.DockerizedTests.BuildId) PROJECT_NAME=DemoProject RELEASE_NAME=Grid TESTSET_NAME=Test docker-compose run automation

where, ${TAG} is assigned the value of $(Release.Artifacts.DockerizedTests.BuildId), ${PROJECT_NAME} is assigned the value “DemoProject”, ${RELEASE_NAME} is assigned the value “Grid” and ${TESTSET_NAME} is assigned the value “Test”. These values can also be passed using Release Variables.

These values are specific inputs for my setup. For you these will invariably be different.

Now lets talk about the second task — “Publish Test Results”. This one is particularly interesting because, the tests are executed inside a container. So the result files are also generated inside the container. However this task expects the result files to be present on the Azure agent. So we need to use the docker volumes to mount a Host Azure agent location on a container location. If we have a relook at the docker-compose.yml file we see that the volumes tag looks like this :

volumes:
      - ./Results/:/workspace/Projects/${PROJECT_NAME}/Results/TestExecution/${RELEASE_NAME}/${TESTSET_NAME}

The way to read this is, “./Results/” location on the Azure agent is mounted on the “/workspace/Projects/${PROJECT_NAME}/Results/TestExecution/${RELEASE_NAME}/${TESTSET_NAME}” location of the container. So as soon as the test results are generated in the container, it is available in the host Azure agent. This can then be used to show Azure Reports.

If we click on the Tests tab after the release pipeline is run, we should see the reports.

You are almost there!! You have executed the tests and generated the reports. The final thing left is to clean up the environment. We would like to stop and remove containers, networks, volumes, and images created by docker-compose run. The best way to do this is :

docker-compose down 

Voila!!

You have successfully set up a complete end to end containerized automated test execution using Azure DevOps.

Happy testing!😊