End-to-end CI/CD automation using Azure DevOps unified Yaml-defined Pipelines - VERY GOOD

Post explains compete automation for CI/CD pipe line. yaml is muti stage which builds and deploys in dev and prod in Azure app services. To execute  code sample follow the few tips given below.

1. GIT:https://github.com/dasaradhreddyk/oss-cicd-devops
2. WEB APP:https://melonbetawebapp11.azurewebsites.net/
3. Instructions
• change resource group and subscription name in yaml fie and location as same as resource group name.
4.  It is very good sample for automating  build  and deploy multistage pipe line.

Azure pipeline as a core part of Azure DevOps, it allows for the creation of CI ( Continuous Integration) pipeline in a declarative way using YAML documents, it is also called build pipelines. Since last Build 2019, this capability is also extending to CD ( Continuous Delivery ) Pipelines which is also known as Release Pipelines.  More than expected is now it is possible to define multi-stage pipelines-as-code for both Continuous Integration and Continuous Delivery with the same YAML definition file. 

Since Github can be easily integrated with Azure DevOps nowadays,  you can not only build your CI/CD pipeline based on your source code on Github, but also even Mapping your GitHub repository permissions with Azure DevOps.

Solution Overview

I am writing this blog to explain how to use Azure CI/CD pipelines to provide an end-to-end automation experience to users when deploying an node.js application via Azure DevOps. Our solution will be look like the following diagram: 

End-toEnd CI/CD architecture diagram

Getting started

Before getting started, you have to prepare a couple of things : 

Step 1: Get an Azure subscription

You need a Azure account : Create an Azure account by browsing to https://azure.microsoft.com/en-us/free/ or claim your MSDN benefits to get a visual studio subscription. 

Step 2: Get an Azure DevOps account and create organisation and project. You can check Quickstart: Create an organization or project collection to know more details.

You’ll need to activate the ‘ multi-stage pipeline’ from the settings of preview feature.

Step 3: Get source from cloudmelon’s Github repository ( using the following URL ), fork it and use your own branch for our lab:

https://github.com/cloudmelon/oss-cicd-devops

Provision infrastructure via ARM template

One of most important part of end-to-end automation start from automate the infrastructure provision process,  by implementing Infrastructure as code (IaC) mindsets, we can use either ARM or Terraform to provision our infrastructure running in Azure.

In this solution, we’re going to provision two App services Plan and host web app on each of them. In enterprise environment, due to budget and performance concerns, usually we deploy Development and Production environment separately. On the top of solution, we can also use full stack monitoring solutions to gain more visibility of Azure resources which we’ll get to later on. 

Where DevOps magic happens : defining service connections

There are so many discussions around automation and DevOps going on over past a few years. From my point of view, one of the most charming feature of DevOps is you can deploy and automate the whole process as long as you have the right permission. In our example, you only need to replace the service connections then you can get this solution up and running in your subscription.  You only need to go to Project settings and add a new service connection as shown in the following : 

Activate the multi-stage pipelines preview feature

Trigger and variable definition

Basically, Continuous integration (CI) triggers in Azure DevOps will set off a build to run when a push is made to the specified branches ( by setting path filters ) or a specified tag is pushed. Below is an example of trigger defined in Yaml:

trigger:
 - master

In Azure DevOps, you can use variable in Yaml definition or using Variable group and then link up to your CI or CD definition. Below is an example of yaml-defined variables:

variables:
 vmImageName: 'ubuntu-latest'
 demorg: 'melon-cicd-rg'
 subscription: '(please replace it by your own service connection name)'
 webappname: 'meloncicdwebapp'

Define your CI pipeline

In your CI pipeline, you need to define as the following :

stage: Build
 displayName: Build stage
 jobs:
job: Build
 displayName: Build
 pool:
   vmImage: $(vmImageName)
 steps:
 task: AzureCLI@1
   displayName: 'Azure CLI '
   inputs:
     azureSubscription: $(subscription)
     scriptLocation: inlineScript
     inlineScript: 'az group create --location north europe --name $(demure)'
 task: AzureResourceGroupDeployment@2
   displayName: 'Azure Deployment:Create or Update Resource Group action on $(demorg)'
   inputs:
     azureSubscription: $(subscription)
     resourceGroupName: '$(demurg)'
     location: 'North Europe'
     templateLocation: 'Linked artifact'
     csmFile: 'iac/webapp.json'
     csmParametersFile: 'iac/webapp.parameters.json'
     deploymentMode: 'Incremental'
 task: NodeTool@0
   inputs:
     versionSpec: '10.x'
     displayName: 'Install Node.js'
     scripts: |
        npm install
        npm run build --if-present npm run test --if-present
     displayName: 'npm install, build and test'
 task: CopyFiles@2
   displayName: 'Copy Files to: $(Build.ArtifactStagingDirectory)/$(webappname)'
   inputs:
     SourceFolder: '$(system.defaultworkingdirectory)'
     TargetFolder: '$(Build.ArtifactStagingDirectory)/$(webappname)'
 task: ArchiveFiles@2
   displayName: '$(webappname) Archive'
   inputs:
     rootFolderOrFile: '$(Build.ArtifactStagingDirectory)/$(webappname)'
     includeRootFolder: false
     archiveType: zip
     replaceExistingArchive: true
     archiveFile: '$(Build.ArtifactStagingDirectory)/$(webappname).zip'
 task: PublishPipelineArtifact@0
   displayName: 'PublishPipelineArtifact: drop'
   inputs:
     targetPath: '$(Build.ArtifactStagingDirectory)/$(webappname).zip'

Define your CD pipeline

Now yaml-defined multi-stage CD pipeline is also supported in Azure DevOps, if you’re not comfortable with yaml definition, you can also check Define your multi-stage continuous deployment (CD) pipeline to know more about multi-stage pipeline. 

Please make sure you define different stages, dependencies and deployment condition:

Dev stage:

- stage Dev
  displayName: Dev stage
  dependsOn: Build
  condition: succeeded('Build')
  jobs:
- deployment: Dev
  displayName: Dev
  environment: 'development'
  pool:
    vmImage: $(vmImageName)
  strategy:
    runOnce: 
      deploy:
        steps:
          - task: DownloadPipelineArtifact@1
            displayName: 'Download Pipeline Artifact'
            inputs:
              buildType: 'current'
          - task: AzureWebApp@1
            inputs:
              azureSubscription: '$(subscription)'
              appType: 'webApp'
              package: '$(System.ArtifactsDirectory)/drop/$(webappname).zip'
              customWebConfig: '-Handler iisnode -NodeStartFile index.js -appType node'
              deploymentMethod: 'zipDeploy'

Prod stage:

- stage: Prod
  displayName: Prod stage
  dependsOn: Dev
  condition: succeeded('Dev')
  jobs:
    - deployment: Prod
      displayName: Prod
      environment: 'production'
      pool:
        vmImage: $(vmImageName)
          strategy:
            runOnce:
              deploy:
                steps:
                - task: DownloadPipelineArtifact@1
                  displayName: 'Download Pipeline Artifact'
          strategy:
            runOnce:
              deploy:
                steps:
                - task: DownloadPipelineArtifact@1
                  displayName: 'Download Pipeline Artifact'
                  inputs:
                    buildType: 'current'
                - task: AzureWebApp@1
                  inputs:
                    azureSubscription: '$(subscription)'
                    appType: 'webApp'
                    appName: '$(prodwebappname)'
                    package: '$(System.ArtifactsDirectory)/drop/$(webappname).zip'
                    customWebConfig: '-Handler iisnode -NodeStartFile index.js -appType node'
                    deploymentMethod: 'zipDeploy'

Environments

Environment is a new feature of Azure DevOps pipeline, it represents a collection of resources such as namespaces within Kubernetes clusters or Azure Web Apps, basically anything be targeted by deployments from a pipeline. In our example environments include Development and Production environment.

Environments in Azure DevOps

Art of possible: Full stack, end-to-end visibility Azure unified monitoring solution 

Since end of 2018, Azure Log Analytics and Azure Application Insights were available as integrated features within Azure Monitor,  it is an pretty cool path which shows the possibility to provide administrators and DevOps engineers full stack unified monitoring solution, you can also check End-to-end monitoring solutions in Azure for Apps and Infrastructure to have more details. 

Up & Running

You can check your run history and different stages in Azure DevOps.

Deploying multi-stage pipelines in Azure DevOps

Perspectives 

The value of end-to-end automation is to cut down time-to-market and to boot your business performance. The scenarios are not only on OSS or .Net application deployment but also make senses on the API economy as well as microservices productivities. However, one of the biggest challenges of this approach is about security and compliance, where DevSecOps comes out, you can find more information about Secure DevOps here.