ASP.NET Core CI/CD on Azure Pipelines with Kubernetes and Helm

 Due to the high entry threshold, it is not that easy to start a journey with Cloud Native. Developing apps focused on reliability and performance, and meeting high SLAs can be challenging. Fortunately, there are tools like Istio which simplify our lives. In this article, we guide you through the steps needed to create CI/CD with Azure Pipelines for deploying microservices using Helm Charts to Kubernetes. This example is a good starting point for preparing your development process. After this tutorial, you should have some basic ideas about how Cloud Native apps should be developed and deployed.

Technology stack

• .NET Core 3.0 (preview)
• Kubernetes
• Helm
• Istio
• Docker
• Azure DevOps

Prerequisites

You need a Kubernetes cluster, free Azure DevOps account, and a docker registry. Also, it would be useful to have kubectl and gcloud CLI installed on your machine. Regarding the Kubernetes cluster, we will be using Google Kubernetes Engine from Google Cloud Platform, but you can use a different cloud provider based on your preferences. On GCP you can create a free account and create a Kubernetes cluster with Istio enabled (Enable Istio checkbox). We suggest using a machine with 3 standard nodes.

Connecting the cluster with Azure Pipelines

Once we have the cluster ready, we have to use kubectl to prepare service account which is needed for Azure Pipelines to authenticate. First, authenticate yourself by including necessary settings in kubeconfig. All cloud providers will guide you through this step. Then following commands should be run:

?

kubectl create serviceaccount azure-pipelines-deploy kubectl create clusterrolebinding azure-pipelines-deploy    --clusterrole=cluster-admin --serviceaccount=default:azure-pipelines-deploy kubectl get secret $(kubectl get secrets -o custom-columns=":metadata.name" | grep azure-pipelines-deploy-token) -o yaml

We are creating a service account, to which a cluster role is assigned. The cluster-admin role will allow us to use Helm without restrictions. If you are interested, you can read more about RBAC on Kubernetes website. The last command is supposed to retrieve secret yaml, which is needed to define connection – save that output yaml somewhere.

Now, in Azure DevOps, go to Project Settings -> Service Connections and add a new Kubernetes service connection. Choose service account for authentication and paste the yaml copied from command executed in the previous step.

One more thing we need in here is the cluster IP. It should be available at cluster settings page, or it can be retrieved via command line. In the example, for GCP command should be similar to this:

?

gcloud container clusters describe --format=value(endpoint) --zone

Another service connection we have to define is for docker registry. For the sake of simplicity, we will use the Docker hub, where all you need is just to create an account (if you don’t have one). Then just supply whatever is needed in the form, and we can carry on with the application part.

Preparing an application

One of the things we should take into account while implementing apps in the Cloud is the Twelve-Factor methodology. We are not going to describe them one by one since they are explained good enough either here or here but few of them will be mentioned throughout the article.

For tutorial purposes, we’ve prepared a sample ASP.NET Core Web Application containing a single controller and database context. It also contains simple dockerfile and helm charts. You can clone/fork sample project from here. Firstly, push it to a git repository (we will use Azure DevOps), because we will need it for CI. You can now add a new pipeline, choosing any of the available YAML definitions. In here we will define our build pipeline (CI) which looks like that:

?

trigger: - master pool:  vmImage: 'ubuntu-latest' variables:  buildConfiguration: 'Release' steps: - task: Docker@2  inputs:    containerRegistry: 'dockerRegistry'    repository: '$(dockerRegistry)/$(name)'    command: 'buildAndPush'    Dockerfile: '**/Dockerfile' - task: PublishBuildArtifacts@1  inputs:    PathtoPublish: '$(Build.SourcesDirectory)/charts'    ArtifactName: 'charts'    publishLocation: 'Container'

Such definition is building a docker image and publishing it into predefined docker registry. There are two custom variables used, which are dockerRegistry (for docker hub replace with your username) and name which is just an image name (exampleApp is our case). The second task is used for publishing artifact with helm chart. These two (docker image & helm chart) will be used for the deployment pipeline.

Helm charts

Firstly, take a look at the file structure for our chart. In the main folder, we have Chart.yaml which keeps chart metadata, requirements.yaml with which we can specify dependencies or values.yaml which serves default configuration values. In the templates folder, we can find all Kubernetes objects that will be created along with chart deployment. Then we have nested charts folder, which is a collection of charts added as a dependency in requirements.yaml. All of them will have the same file structure.

Let’s start with a focus on the deployment.yaml – a definition of Deployment controller, which provides declarative updates for Pods and Replica Sets. It is parameterized with helm templates, so you will see a lot of {{ template […] }} in there. Definition of this Deployment itself is quite default, but we are adding a reference for the secret of SQL Server database password. We are hardcoding ‘-mssql-linux-secret’ part cause at the time of writing this article, helm doesn’t provide a straightforward way to access sub-charts properties.

?

env: - name: sa_password   valueFrom:     secretKeyRef:       name: {{ template "exampleapp.name" $root }}-mssql-linux-secret       key: sapassword

As we mentioned previously, we do have SQL Server chart added as a dependency. Definition of that is pretty simple. We have to define the name of the dependency, which will match the folder name in charts subfolder and the version we want to use.

?

dependencies: - name: mssql-linux   repository: https://kubernetes-charts.storage.googleapis.com   version: 0.8.0   [...]

For the mssql chart, there is one change that has to be applied in the secret.yaml. Normally, this secret will be created on each deployment (helm upgrade), it will generate a new sapassword – which is not what we want. The simplest way to adjust that is by modifying metadata and adding a hook on pre-install. This will guarantee that this secret will be created just once on installing the release.

?

metadata:   annotations:     "helm.sh/hook": "pre-install"

A deployment pipeline

Let’s focus on deployment now. We will be using Helm to install and upgrade everything that will be needed in Kubernetes. Go to the Releases pipelines on the Azure DevOps, where we will configure continuous delivery. You have to add two artifacts, one for docker image and second for charts artifact. It should look like on the image below.

On the stages part, we could add a few more environments, which would get deployed in a similar manner, but to a different cluster. As you can see, this approach guarantees Deploy DEV stage is simply responsible for running a helm upgrade command. Before that, we need to install helm, kubectl and run helm init command.

For the helm upgrade task, we need to adjust a few things.

• set Chart Path, where you can browse into Helm charts artifact (should look like: “$(System.DefaultWorkingDirectory)/Helm charts/charts”)
• paste that “image.tag=$(Build.BuildNumber)” into Set Values
• and check to Install if release not present or add –install ar argument. This will behave as helm install if release won’t exist (i.e. on a clean cluster)

At this point, we should be able to run the deployment application – you can create a release and run deployment. You should see a green output at this point :).

You can verify if the deployment went fine by running a kubectl get all command.

Making use of basic Istio components

Istio is a great tool, which simplifies services management. It is responsible for handling things like load balancing, traffic behavior, metric & logs, and security. Istio is leveraging Kubernetes sidecar containers, which are added to pods of our applications. You will have to enable this feature by applying an appropriate label on the namespace.

?

kubectl label namespace default istio-injection=enabled

All pods which will be created now will have an additional container, which is called a sidecar container in Kubernetes terms. That’s a useful feature, cause we don’t have to modify our application.

Two objects that we are using from Istio, which are part of the helm chart, are Gateway and VirtualService. For the first one, we will bring Istio definition, because it’s simple and accurate: “Gateway describes a load balancer operating at the edge of the mesh receiving incoming or outgoing HTTP/TCP connections”. That object is attached to the LoadBalancer object – we will use the one created by Istio by default. After the application is deployed, you will be able to access it using LoadBalancer external IP, which you can retrieve with such command:

?

kubectl get service/istio-ingressgateway -n istio-system

You can retrieve external IP from the output and verify if http://api/examples url works fine.

Summary

In this article, we have created a basic CI/CD which deploys single service into Kubernetes cluster with the help of Helm. Further adjustments can include different types of deployment, publishing tests coverage from CI or adding more services to mesh and leveraging additional Istio features. We hope you were able to complete the tutorial without any issues. Follow our blog for more in-depth articles around these topics that will be posted in the future.

Azure DevOps Environments for Kubernetes EXPLAINED

 Azure Pipelines is a great tool for doing Continuous Integration and Continuous Deployment and thanks to Multi Stage Pipelines we can finally have build, test, and release directly expressed in source code.

Recently they have also introduced the concept of "Environments", which belongs to the release process.

In the previous article of this series we've looked at the Environments in general.

Today, instead, we will go deeper into it and we will explore the integration between environments and Kubernetes clusters.

In fact, the Kubernetes resource is the first that has been made available in Azure DevOps Environments and it's the one which provides the most features.

Let's see first what we can do with it and then how to set this up.

Video

If you are a visual learner or simply prefer to watch and listen instead of reading, here you have the video with the whole explanation, which to be fair is much more complete than this post.

If you rather prefer reading, well... let's just continue :)

Deployments

When you click on the environment, you have 2 separate tabs: Resources and Deployments.

The Deployments tab has the same content we've seen in the previous post about Environments in general, so I won't spend much time about it.

Just real quick, in here you can see what has been deployed and when.

For example here I can see that this deployment came from the deploy job of the K8S_CICD Pipelines, ran on May 25.

Also, if I go inside, I can see what changes are included (and I have visibility down to the single file diffs):

and that all of this was initially planned in the linked work items, here for example a bug and a task:

Thanks to the complete integration of all the parts of Azure Devops, we have full traceability from Work management to deployments, and everything in between.

But as we have seen in the previous article, this comes with any kind of Environment, not only with the Kubernetes ones. So let's see what is unique about this one.

Resources

Back in the Environment page, the Resources tab is where the magic happens.

In fact here I can basically explore the content of my Kubernetes cluster!

Again, we have two tabs here: Workloads and Services.

Workloads lists the Deployments with their Replica Sets:

You can drill into the Replica Set to see its details and the Pods it's running on:

Note that here you can see not only the image associated with this deployment, but also its label and selectors (if any).

Finally, drilling into a pod, you can see its full details:

But that's not all, you can go even deeper. In fact, we can see the Logs the Pod is producing:

and even the full YAML of the Pod coming directly from K8S!

If you are curious about the YAML content, this is what you get:

apiVersion: v1
kind: Pod
metadata:
  name: webserver-7f6cf4c486-4mtcj
  generateName: webserver-7f6cf4c486-
  namespace: default
  selfLink: /api/v1/namespaces/default/pods/webserver-7f6cf4c486-4mtcj
  uid: 536bafd1-9d45-44d3-b674-3e0d91345d1c
  resourceVersion: '899183'
  creationTimestamp: '2020-05-19T06:17:44Z'
  labels:
    app: nginx
    pod-template-hash: 7f6cf4c486
  annotations:
    azure-pipelines/jobName: '"Deploy"'
    azure-pipelines/org: 'https://dev.azure.com/dbtek/'
    azure-pipelines/pipeline: '"K8S_CICD"'
    azure-pipelines/pipelineId: '"65"'
    azure-pipelines/project: AKSEnvironmentDemo
    azure-pipelines/run: '20200525.1'
    azure-pipelines/runuri: 'https://dev.azure.com/dbtek/AKSEnvironmentDemo/_build/results?buildId=1058'
    cni.projectcalico.org/podIP: 10.244.1.9/32
  ownerReferences:
    - apiVersion: apps/v1
      kind: ReplicaSet
      name: webserver-7f6cf4c486
      uid: 1daf0d4e-5daf-4926-a4ee-42c735fb0071
      controller: true
      blockOwnerDeletion: true
spec:
  volumes:
    - name: default-token-mcfjv
      secret:
        secretName: default-token-mcfjv
        defaultMode: 420
  containers:
    - name: nginx
      image: 'nginx:1.17.10'
      ports:
        - containerPort: 80
          protocol: TCP
      resources: {}
      volumeMounts:
        - name: default-token-mcfjv
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
  restartPolicy: Always
  terminationGracePeriodSeconds: 30
  dnsPolicy: ClusterFirst
  serviceAccountName: default
  serviceAccount: default
  nodeName: aks-agentpool-17828392-vmss000002
  securityContext: {}
  schedulerName: default-scheduler
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
  priority: 0
  enableServiceLinks: true
status:
  phase: Running
  conditions:
    - type: Initialized
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2020-05-19T06:17:44Z'
    - type: Ready
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2020-05-19T06:17:56Z'
    - type: ContainersReady
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2020-05-19T06:17:56Z'
    - type: PodScheduled
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2020-05-19T06:17:44Z'
  hostIP: 10.240.0.6
  podIP: 10.244.1.9
  podIPs:
    - ip: 10.244.1.9
  startTime: '2020-05-19T06:17:44Z'
  containerStatuses:
    - name: nginx
      state:
        running:
          startedAt: '2020-05-19T06:17:56Z'
      lastState: {}
      ready: true
      restartCount: 0
      image: 'nginx:1.17.10'
      imageID: >-
        docker-pullable://nginx@sha256:30dfa439718a17baafefadf16c5e7c9d0a1cde97b4fd84f63b69e13513be7097
      containerID: >-
        docker://6c8ebc8692033d4a0e92fe247fa4970ed77d664876cf42a7b8f5bad012eb2c68
      started: true
  qosClass: BestEffort

I love it, it's so useful and having everything in one tool allows you to focus much more!

Now that we have explored what we can do with it, let's see how to create a new Environment for Kubernetes in Azure Pipelines.

Create a Kubernetes environment

First of all, needless to say, you need to have a Kubernetes cluster. I always use AKS in Azure because is a managed service and I don't have to pay for the master nodes ;)

And using AKS is also much easier linking the cluster with Azure DevOps.

First of all, let's go to the Environments section under Pipelines, and click "New Environment".

Let's give the environment a name and select "Kubernetes" as type of Resource.

Here you can set whether you want to use AKS or any other Kubernetes cluster. If you go for a Generic Kubernetes, then you'll need to input all the parameters manually and set up your cluster so it is reachable from Azure DevOps.

But if you select AKS, you'll get prompted with the selection directly.

Just pick the Azure Subscription and the cluster, and select the namespace you want to link. In fact, each resource maps to a specific Namespace in your cluster.

If you already have an environment linked to your default namespace, you can create a new one.

For this post I decided to call the new Environment Secondary and to create a new namespaces called app2ns.

This of course works if the account with which you're accessing Azure Devops has the proper permissions in your Azure subscription.

And that's it! Of course, since we've just created it, it is completely empty.

Now all we have to do is using this environment in a Deployment job of a pipeline to have all the goodness we've seen before.

Let's use it in a Pipeline

As mentioned, you need to have a pipeline which uses a Deployment Job, not a "normal one".

Then add the environment to it. The format to use is "EnvironmentName dot Namespace". In my case, this is Secondary.app2namespace

This is the snippet:

- stage: CD
  displayName: CD Stage
  dependsOn: CI
  jobs:

  - deployment: deploy
    displayName: Deploy
    environment: Secondary.app2namespace
    strategy: 
      runOnce:
        deploy:
          steps: 

          - task: KubernetesManifest@0
            inputs:
              action: 'deploy'
              manifests: '$(Pipeline.Workspace)/YAML Files/k8s/App2.yaml'

The beauty of this, looking down at the task which actually perform the deployment, is that we do not need to specify any service connection, credentials, or anything else.

The reason for this is that the Pipelines engine will take everything it needs from the Environment directly.

If you then make a change at your code, this will start the whole process of CI CD, and bring together all the componentes: code, work items, build and deployment.

Conclusion

Cool right? I mean, there are cooler things than this but they are all in the outside world... like, the real world... you know what I mean :D

Alright, this is how to use the Azure DevOps Environments for Kubernetes and how amazing and useful it is working this way.

Examples

Take a look at my video on YoutTube here to see how to create, manage, and use the Environments for Kubernetes in Azure DevOps.