Saturday, July 31, 2021

Azure DevOps Multi-Stage Pipelines

Summary :  Very simple article to push docker to ACR and use in web app
1. create resource group 

az group create --name $(APP_NAME)-rg --location $(LOCATION)

2.launches an Azure Container Registry if one doesn’t exist,

az acr create --resource-group $(APP_NAME)-rg --name $(ACR_NAME) --sku Basic --admin-enabled true

3. Push image to ACR

export THETIMEANDDATE=$(date '+%Y-%m-%d-%H%M')

echo "$THETIMEANDDATE will be the point in time tag"

az acr login --name $(ACR_NAME) 

docker image build -t $(IMAGE_NAME) ./

docker image tag $(IMAGE_NAME) $(ACR_NAME).azurecr.io/$(IMAGE_NAME):latest

docker image tag $(IMAGE_NAME) $(ACR_NAME).azurecr.io/$(IMAGE_NAME):$THETIMEANDDATE

docker image push $(ACR_NAME).azurecr.io/$(IMAGE_NAME):latest 

docker image push $(ACR_NAME).azurecr.io/$(IMAGE_NAME):$THETIMEANDDATE

4 configures and launches an App Service Plan, App Service and then configures automatic deployment 

az webapp config container set --resource-group $(APP_NAME)-rg --name $(APP_NAME)

--docker-custom-image-name $(ACR_NAME).azurecr.io/$(IMAGE_NAME):latest

--docker-registry-server-url https://$(ACR_NAME).azurecr.io

az webapp deployment container config

--resource-group $(APP_NAME)-rg

--name $(APP_NAME)

--enable-cd true

 As I move into using Azure DevOps more and more, there is one thing which I really haven’t got on with and that is release pipelines. I didn’t like that I couldn’t easily define it as YAML like I could with build pipelines, even though I don’t like them, there are a few things I do like — like approval gates.

 

Environments

Luckily, there are ways to get around this — the most simple way is to add an Environment and then add an Approval. Adding an Environment is simple enough, just click on Environment in your Pipeline and then add one with None selected under Resource;

Adding an environment {Source: MediaGlasses}

 

Once you have added the Environment you can then add an approval check, to do this click on the Environment you have just created and click on the three dots in the top right-hand side of the page. From the menu select Approvals and checks;

Adding an approval {Source: MediaGlasses}

 

Now we have an Environment and the Approval in place we can move onto the Pipeline.

 

Pipeline

- stage: "SetupRG"
displayName: "Resource Group"
jobs:
- job: "CreateResourceGroup"
displayName: "Resource Group - Setup"
steps:
- task: AzureCLI@2
inputs:
azureSubscription: "$(SUBSCRIPTION_NAME)"
addSpnToEnvironment: true
scriptType: "bash"
scriptLocation: "inlineScript"
inlineScript: |
az group create --name $(APP_NAME)-rg --location $(LOCATION)
displayName: "Resource Group - Use Azure CLI to setup or check"
view raw01.yml hosted with ❤ by GitHub
- stage: "SetupACR"
displayName: "Azure Container Registry"
dependsOn:
- "SetupRG"
jobs:
- job: "SetupCheckAzureContainerRegistry"
displayName: "Azure Container Registry - Setup"
variables:
- name: "DECODE_PERCENTS"
value: true
steps:
- task: AzureCLI@2
inputs:
azureSubscription: "$(SUBSCRIPTION_NAME)"
addSpnToEnvironment: true
scriptType: "bash"
scriptLocation: "inlineScript"
inlineScript: |
ACR_ID=$(az acr show --resource-group $APP_NAME-rg --name $ACR_NAME --query "id" -o tsv)
if [ -z "$ACR_ID" ]; then
echo "There is no Azure Container Registry, we should sort that"
az acr create --resource-group $(APP_NAME)-rg --name $(ACR_NAME) --sku Basic --admin-enabled true
else
echo "There is already an Azure Container Registry, we don't need to do anything else here"
fi
displayName: "Azure Container Registry - Use Azure CLI check or setup"
view raw02.yml hosted with ❤ by GitHub
Now that we have a container registry to push our image to we can build and the push the container, this is the stage where we will be getting approval before building;
- stage: "BuildContainer"
displayName: "Build, Tag and Push the container image"
dependsOn:
- "SetupACR"
jobs:
- deployment: BuildPushImage
displayName: "Build, tag and push the image"
environment: "production"
pool:
vmImage: "Ubuntu-20.04"
strategy:
runOnce:
deploy:
steps:
- checkout: self
- task: AzureCLI@2
inputs:
azureSubscription: "$(SUBSCRIPTION_NAME)"
addSpnToEnvironment: true
scriptType: "bash"
scriptLocation: "inlineScript"
inlineScript: |
export THETIMEANDDATE=$(date '+%Y-%m-%d-%H%M')
echo "$THETIMEANDDATE will be the point in time tag"
az acr login --name $(ACR_NAME)
docker image build -t $(IMAGE_NAME) ./
docker image tag $(IMAGE_NAME) $(ACR_NAME).azurecr.io/$(IMAGE_NAME):latest
docker image tag $(IMAGE_NAME) $(ACR_NAME).azurecr.io/$(IMAGE_NAME):$THETIMEANDDATE
docker image push $(ACR_NAME).azurecr.io/$(IMAGE_NAME):latest
docker image push $(ACR_NAME).azurecr.io/$(IMAGE_NAME):$THETIMEANDDATE
displayName: "Use Azure CLI to build and push the container image"
view raw03.yml hosted with ❤ by GitHub
- stage: "SetupAppServices"
displayName: "Azure App Services"
dependsOn:
- "SetupRG"
- "SetupACR"
- "BuildContainer"
jobs:
- job: "CheckForAppServicePlan"
displayName: "App Service Plan - Check if App Service Plan exists"
steps:
- task: AzureCLI@2
inputs:
azureSubscription: "$(SUBSCRIPTION_NAME)"
addSpnToEnvironment: true
scriptType: "bash"
scriptLocation: "inlineScript"
inlineScript: |
APP_SERVICE_PLAN_ID=$(az appservice plan show --resource-group $APP_NAME-rg --name $APP_NAME-asp --query "id" -o tsv)
if [ -z "$APP_SERVICE_PLAN_ID" ]; then
echo "There is no App Service Plan, we should sort that"
echo "##vso[task.setvariable variable=appServiceExist;isOutput=true]No" # there is no app service plan so we should do stuff
else
echo "There is an App Service Plan, we don't need to do anything else here"
echo "##vso[task.setvariable variable=appServiceExist;isOutput=true]Yes" # nothing to do lets move on
fi
name: "DetermineResult"
displayName: "App Service Plan - Check to see if there App Service Plan exists"
- job: "CreateAppServicePlan"
displayName: "App Service Plan - Setup"
dependsOn:
- "CheckForAppServicePlan"
condition: "eq(dependencies.CheckForAppServicePlan.outputs['DetermineResult.appServiceExist'], 'No')"
steps:
- task: AzureCLI@2
inputs:
azureSubscription: "$(SUBSCRIPTION_NAME)"
addSpnToEnvironment: true
scriptType: "bash"
scriptLocation: "inlineScript"
inlineScript: |
az appservice plan create --resource-group $(APP_NAME)-rg --name $(APP_NAME)-asp --is-linux
displayName: "App Service Plan - Use Azure CLI to setup"
- job: "CreateAppService"
displayName: "Web App - Setup"
dependsOn:
- "CheckForAppServicePlan"
- "CreateAppServicePlan"
condition: "eq(dependencies.CheckForAppServicePlan.outputs['DetermineResult.appServiceExist'], 'No')"
steps:
- task: AzureCLI@2
inputs:
azureSubscription: "$(SUBSCRIPTION_NAME)"
addSpnToEnvironment: true
scriptType: "bash"
scriptLocation: "inlineScript"
inlineScript: |
az webapp create --resource-group $(APP_NAME)-rg --plan $(APP_NAME)-asp --name $(APP_NAME) --deployment-container-image-name $(ACR_NAME).azurecr.io/$(IMAGE_NAME):latest
displayName: "Web App - Use Azure CLI to setup"
- job: "CreateAppServiceSettings"
displayName: "Web App - Configure Settings"
dependsOn:
- "CheckForAppServicePlan"
- "CreateAppServicePlan"
- "CreateAppService"
condition: "eq(dependencies.CheckForAppServicePlan.outputs['DetermineResult.appServiceExist'], 'No')"
steps:
- task: AzureCLI@2
inputs:
azureSubscription: "$(SUBSCRIPTION_NAME)"
addSpnToEnvironment: true
scriptType: "bash"
scriptLocation: "inlineScript"
inlineScript: |
az webapp config appsettings set --resource-group $(APP_NAME)-rg --name $(APP_NAME) --settings $(APP_SETTINGS)
displayName: "Web App - Use Azure CLI to configure the settings"
- job: "CreateAppServiceID"
displayName: "Web App - Configure & Assign Managed Identity"
dependsOn:
- "CheckForAppServicePlan"
- "CreateAppServicePlan"
- "CreateAppService"
- "CreateAppServiceSettings"
condition: "eq(dependencies.CheckForAppServicePlan.outputs['DetermineResult.appServiceExist'], 'No')"
steps:
- task: AzureCLI@2
inputs:
azureSubscription: "$(SUBSCRIPTION_NAME)"
addSpnToEnvironment: true
scriptType: "bash"
scriptLocation: "inlineScript"
inlineScript: |
az webapp identity assign --resource-group $(APP_NAME)-rg --name $(APP_NAME)
displayName: "Web App - Use Azure CLI to assign an identity"
- task: AzureCLI@2
inputs:
azureSubscription: "$(SUBSCRIPTION_NAME)"
addSpnToEnvironment: true
scriptType: "bash"
scriptLocation: "inlineScript"
inlineScript: |
az role assignment create --assignee $(az webapp identity show --resource-group $(APP_NAME)-rg --name $(APP_NAME) --query principalId --output tsv) --scope /subscriptions/$(az account show --query id --output tsv)/resourceGroups/$(APP_NAME)-rg/providers/Microsoft.ContainerRegistry/registries/$(ACR_NAME) --role "AcrPull"
displayName: "Web App - Use Azure CLI to assign an identity"
- job: "EnableCD"
displayName: "Web App - Configure Azure Container Registry connedction and enable continuous deployment"
dependsOn:
- "CheckForAppServicePlan"
- "CreateAppServicePlan"
- "CreateAppService"
- "CreateAppServiceSettings"
- "CreateAppServiceID"
condition: "eq(dependencies.CheckForAppServicePlan.outputs['DetermineResult.appServiceExist'], 'No')"
steps:
- task: AzureCLI@2
inputs:
azureSubscription: "$(SUBSCRIPTION_NAME)"
addSpnToEnvironment: true
scriptType: "bash"
scriptLocation: "inlineScript"
inlineScript: |
az webapp config container set --resource-group $(APP_NAME)-rg --name $(APP_NAME) --docker-custom-image-name $(ACR_NAME).azurecr.io/$(IMAGE_NAME):latest --docker-registry-server-url https://$(ACR_NAME).azurecr.io
displayName: "Web App - Configure the App Serivce to use Azure Container Registry"
- task: AzureCLI@2
inputs:
azureSubscription: "$(SUBSCRIPTION_NAME)"
addSpnToEnvironment: true
scriptType: "bash"
scriptLocation: "inlineScript"
inlineScript: |
az webapp deployment container config --resource-group $(APP_NAME)-rg --name $(APP_NAME) --enable-cd true
displayName: "Web App - Enable continuous deployment whenever the image is updated on the WebApp"
- task: AzureCLI@2
inputs:
azureSubscription: "$(SUBSCRIPTION_NAME)"
addSpnToEnvironment: true
scriptType: "bash"
scriptLocation: "inlineScript"
inlineScript: |
az acr webhook create --name $(ACR_NAME)webhook --registry $(ACR_NAME) --scope $(IMAGE_NAME):latest --actions push --uri $(az webapp deployment container show-cd-url --resource-group $(APP_NAME)-rg --name $(APP_NAME) --query "CI_CD_URL" -o tsv)
displayName: "Azure Container Registry - Add the web hook created in the last task to the Azure Container Registry"
view raw04.yml hosted with ❤ by GitHub

A full copy of the azure-pipelines.yml file can be found in the following repo https://github.com/russmckendrick/docker-python-web-app.

 

Running the Pipeline

Now that we know what the pipeline looks like this what happened when it executed for the first time, first off, you (or whoever your approver is) will get an email;

The approval email {Source: MediaGlasses}

 

The full pipeline run {Source: MediaGlasses
A second pipeline run {Source: MediaGlasses}

 

Azure DevOps Multi-Stage Pipelines

 Treating your infrastructure as code is becoming more and more necessary these days. Writing these instructions becoming challenging too. In Azure we use ARM templates to define the resources and associate them with a deployment pipeline. But ARM templates are quite complicated and they are not everybody’s cup of tea.

Azure Bicep tries to resolve that and, after using it for a while I am so excited to do Bicep templates whenever I can. The dev experience using the bicep templates are highly satisfying.

What is Bicep?

Bicep is a domain specific language which declares your Azure resources to be deployed. It provides uncomplicated syntaxes and have been designed to reuse the code very easily through modules.

ARM templates Vs Bicep

We all have worked with ARM templates and one of its main “challenges” are depending on the resource you deploy you will need to know exactly what to configure and, this can be quite frustrating. With Bicep, the syntax is very concise. You just declare the resource and that’s that.

It’s kind of you going to a pizza shop and mentioning what’s the pizza you would like to have, the size of it, and the toppings. You don’t tell them how to make the pizza dough, or how to pick the jalapenos or which supermarket you want the meat from.

See the below Bicep template which will declare a consumption based app service.

The same app service plan done using ARM template will look like below

The Bicep template is very straight forward, where as the ARM template syntax you need to be very explicit about what you require.

Bicep playground

A good starting point will be the Bicep playground. In there you can experiment with Bicep. The most important feature I like in there is that the ability to transform your existing ARM templates into Bicep . You can simply click the “Decompile” button and point to the ARM template. But be mindful that all your ARM templates might not be easily converted into Bicep templates. You just need to fix the errors which it will show you so generously.

Required tools

  • VSCode
  • Bicep extension for VSCode

Absolutely love the Bicep extension! It has code snippets, syntax highlighting and, even intellisense!

You will find these features highly useful when building Bicep templates.

Resources required to deploy the function app

In here we are planning to deploy an Azure function app. First we’ll identify what are the resources required and create them using Bicep templates. Then like LEGO we’ll use these small building blocks to create the final template which we can use to deploy.

We will be using YAML based multi stage Azure DevOps pipeline to create the build and deployment pipeline.

To deploy an Azure function app you will need the below resources.

  • Resource group
  • Storage account
  • App service plan
  • App insights (optional but highly recommended)
  • Key Vault (optional)
  • Function App and its settings

Storage account

Let’s start with the storage account.

The storage account is so straight forward. You get the storage account kind and the storage account tier as parameters and use them to declare the storage account you require. Finally in the output section it outputs the connection string to the storage account.

Application service plan

Every function app needs to be hosted. Most of the time this application service plan is already created and will be shared among other applications. But for this scenario I wanted to create a separate consumption based app service plan.

Application insights

Although this component is optional it’s highly recommended. The reason being you will be able to see your function app’s performance, failures in one central place.

Function app (without the settings)

First let’s create the skeleton function app without the settings. The key vault (next step) will need to know the function app’s principalid and the tenantid to provide access.

The function app which we’ll be creating will use deployment slots (azure function apps supports production and staging slots only).

So the below Bicep template creates function app with the two slots. Since the AKV will need the principalid and the tenantid the template will output them.

Key vault

Although a key vault can be optional, in reality when developing applications most of the time we’ll have some settings which we would like to make them securely accessible. You can use an Azure key vault to securely save and version your secrets. The below template is accepting the function app’s principalid and the tenantid to setup the access policies.

Finally it outputs the URI (the latest version) of the secret to be used.

Function app settings

Finally let’s set up the configurations required for the function app.

Using these building blocks together

Now we have all the building blocks to create the function app. Bicep has this cool feature where you can create modules. Now lets use Bicep modules to organize our resources to be deployed.

This is the storage account module

As you can see the syntax is very easy and straight forward. You define the module using the module keyword and the location for the Bicep template. Then you just simply pass the parameters required for the template. The Bicep extension of VSCode really helps you out here. As shown below the full intellisense is provided to you. It even give you a bunch of options including conditional access and my personal favourite required-properties .

But when deploying the Azure resources you will need to know the dependencies between them when deploying. In our function app these are the dependencies when the Azure resources need to be deployed.

  • Function app has dependencies on the storage account and the application service plan.
  • The key vault has a dependency on the function app because it needs to provide access to the function app.
  • The function app settings module has dependencies on function app, app insights and the key vault module.
  • The storage account, application service plan and the app insights modules does not have any dependencies.

The dependencies among the Bicep modules are specified as DependsOn. The function app module dependencies can be defined as shown below.

Azure DevOps pipelines

Lets build a YAML based multi-stage pipeline to build and deploy our function app.

Build pipeline

The main purpose of a build pipeline is to check whether your code can be build successfully and to be able to create artifacts.

These are the steps associated with the build pipeline above,

  • Build and restore your .NET projects.
  • Run the tests.
  • Create the function app artifact and package it as a zip.
  • Create the Bicep templates as an artifact so that they can be referenced easily in the deployment pipeline.

Deployment pipeline

The steps involved in the deployment pipeline are as follows,

  • Create the resource group.

We’ll use AZURE CLI to create the resource group.

Creating the resource group using Azure CLI

The resource group is created only if it doesn’t exist.

  • Provision the resources

We have the main Bicep template which orchestrates all the required resources to deployed in Azure. You can use the same Azure CLI command to deploy resources, az deployment group create here.

Passing parameters to the main bicep template to provision resources

Notice the --template-file argument. Since we created all the Bicep templates as an artefact with the name deploy (see the build pipeline above) we can easily use it to locate the Bicep template. Then all the parameters which are required by the main.bicep template is passed.

  • Deploy to the staging slot (stop + deploy latest code + start)

Since we are deploying an HTTP triggered function app with slots, we need to deploy our code to the staging slot first. Let’s stop it first, secondly we’ll deploy the latest code there and then start the staging slot.

  • Deploy to the production slot (swap with staging + stop staging slot)
swap staging slot with production

Since the staging slot is up and running, we can perform the swap operation with the production slot as specified above. Once it’s done no need to keep the staging slot alive. So as the final step we can stop the staging slot.

The build and deployment pipeline

Now we have a build pipeline and a deployment pipeline. In reality the applications we develop will be deployed in multiple environments. So let’s create the final piece where we can build and deploy the function app into multiple environments.

  • Trigger points

We’ll trigger the pipeline when the code is checked in to the master branch or to any branch under feature . Also for PR requests made comparing the code to the master branch (you can make these conditional for each environment as well).

As you can see above there are multiple stages in the pipeline,

BUILD -> DEV -> SITare the stages in this pipeline, but you can create other environments as stages in the pipeline as you wish. Notice how the variable files are placed. There’s one common.yaml file to store all the parameters and each [env].yaml file to override the parameter values which will be specific to the environment.

Setting up the pipeline in Azure DevOps

Create a new pipeline in Azure DevOps, but since you already have the files required please select the file which has instructions to build and deploy your solution before you finish creating the pipeline in Azure DevOps.

Select the existing pipeline file in your solution.

After running the pipeline, you will be able to see the resources successfully deployed in Azure.

The Azure DevOps pipeline

The created resource groups in Azure

The created resource groups

The deployment histories in resource groups.

Finally, the resources deployed in the resource group.

The resources deployed in the resource group

Conclusion

Azure Bicep is awesome! The feature which I like most is the module support. VSCode and the Bicep extension is really helpful when you are building templates.

The code for this can be found in our GitHub

References

https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/overview

Free hosting web sites and features -2024

  Interesting  summary about hosting and their offers. I still host my web site https://talash.azurewebsites.net with zero cost on Azure as ...