CI/CD Docker Compose using Azure DevOps

Summary:

Excellent article with multi stage azure dev ops pipe line created to push dcker image to VM and docker compose used to deploy multiple containers to VM though SSH component .

• Stage 1 is BUILD_docker_containers: Here build and pushing artifacts to ACR explained.
• Stage 2 is DEPLOY_to_production:SSH deployment task used for this.
• As last stage of deployment multiple containers to prodution application and DB containers has been plugged together with docker compose file. 

I've been fighting a bit with some of the Azure DevOps pipeline tasks trying to configure end-to-end solution for one of my side project. It is based on a good old Docker Compose and I am pretty happy with how it works in production. What I wanted to do is schematically described down below.

What is Azure DevOps

Azure DevOps helps to plan smarter, collaborate better, and ship faster with a set of modern dev services. It's a end-to-end solution for any software development cycle. Anyone can use it even for free with some limitations/conditions of usage (public projects, limited pipeline minutes per month etc). And it's free unlimited git!

Multistage pipeline

If you are not familiar with multistage pipeline concept, have a look. In short, it brings all CI/CD experience into yaml, where you define all your stages (no UI.. ). It's been really big missing thing for a while since only build pipeline could be explain like this..

stages:
  - stage: Build_docker_containers
    jobs:
    - job: Build
      pool:
        vmImage: 'Ubuntu-16.04'
      continueOnError: true
      steps:
      - task: Docker@2
        inputs:
          containerRegistry: 'AZURE-CONTAINER-REGISTRY-NAME'
          repository: 'AZURE-CONTAINER-REGISTRY-REPOSITORY-NAME'
          command: 'buildAndPush'
          Dockerfile: '**/Dockerfile'
      - task: PublishPipelineArtifact@1
        inputs:
          targetPath: '$(Pipeline.Workspace)'
          artifact: 'docker-compose'
          publishLocation: 'pipeline'
  
  - stage: 'Deploy_to_production'
    jobs:
    - deployment: Production
      pool:
        vmImage: 'Ubuntu-16.04'
      environment: 'Production'
      strategy:
        runOnce:
          deploy:
            steps:
            - task: CopyFilesOverSSH@0
              inputs:
                sshEndpoint: 'SSH-END-POINT-NAME-FROM-SERVICE-CONNECTIONS'
                sourceFolder: '$(Pipeline.Workspace)/docker-compose/s/'
                contents: |
                  docker-compose.yaml
                  .env
                targetFolder: 'TARGET-PATH'
            - task: SSH@0
              inputs:
                sshEndpoint: 'SSH-END-POINT-NAME-FROM-SERVICE-CONNECTIONS'
                runOptions: 'inline'
                inline: |
                  sed -i 's/##BUILD##/$(Build.BuildId)/g' docker-compose.yaml
            - task: SSH@0
              inputs:
                sshEndpoint: 'SSH-END-POINT-NAME-FROM-SERVICE-CONNECTIONS'
                runOptions: 'inline'
                inline: |
                  docker-compose up -d 2> docker-compose.log
                  cat docker-compose.log

Let's break down the above into small parts and explain what was going on there. In the first stage Build_docker_containers there are two tasks: build image (it's actually three actions in one: build, tag and push) and publish pipeline artifact. Use this task in a pipeline to publish artifacts for the Azure Pipeline (note that publishing is NOT supported in release pipelines. It is supported in multi stage pipelines, build pipelines, and yaml pipelines). AZURE-CONTAINER-REGISTRY-NAME and AZURE-CONTAINER-REGISTRY-REPOSITORY-NAME both have to be changed accordingly. Note that a built image gets tag which is by default built-in Build.BuildId predefined variable which helps me properly roll out my app update on the second stage.

For the sake of simplicity this pipeline (stages) has been simplified.

The second stage is Deploy_to_production and it rolls out built image to my production server. All it's tasks are based on SSH deployment tasks. The SSH endpoint has to be configured first in service endpoints (there used to be some issues in new service endpoint experience with >2048 public keys in 2019, but Microsoft team has fixed this).

sed -i 's/##BUILD##/$(Build.BuildId)/g' docker-compose.yaml replaces build number, so docker-compose up -d 2> docker-compose.log brings something to update in docker-machine.

Docker and Docker Compose

I use docker for packaging an app and docker registry (Azure Container Registry). There is no problem to use dockerhub, just a corresponding endpoint has to be present in service endpoints. Compose helps me to combine multiple containers and define the logic between them and some other objects, as well as their behavior.

Docker compose for Azure DevOps

The following docker-compose.yaml is in my project.

version: '3'
volumes:
  postgres_data: {}
services:
  app:
    image: AZURE-CONTAINER-REGISTRY-NAME.azurecr.io/AZURE-CONTAINER-REGISTRY-REPOSITORY-NAME:##BUILD##
    restart: always
    environment: 
      RAILS_SERVE_STATIC_FILES: 'true'
      COMPOSE_PROJECT_NAME: 'PROJECT-PREFIX'
    depends_on:
      - db
  db:
    restart: always
    image: postgres
    volumes:
      - postgres_data:/var/lib/postgresql/data

Let's break down the above into small parts and explain what was going on there. There are two services (of course there are more, but for the sake of simplicity of this exercise there are only two). AZURE-CONTAINER-REGISTRY-NAME.azurecr.io/AZURE-CONTAINER-REGISTRY-REPOSITORY-NAME:##BUILD## has to be changes slightly except ##BUILD## which is being changed every pipeline execution.

Summary

Any feature or bug fix can be delivered to production in less than 6 minutes.

With 1800 free minutes of pipeline per month and 6 minutes of all my stages total duration I can do 300 cycles building and releasing my software.

Love it!