Azure DevOps Multi-Stage Pipelines

 Treating your infrastructure as code is becoming more and more necessary these days. Writing these instructions becoming challenging too. In Azure we use ARM templates to define the resources and associate them with a deployment pipeline. But ARM templates are quite complicated and they are not everybody’s cup of tea.

Azure Bicep tries to resolve that and, after using it for a while I am so excited to do Bicep templates whenever I can. The dev experience using the bicep templates are highly satisfying.

‍

What is Bicep?

Bicep is a domain specific language which declares your Azure resources to be deployed. It provides uncomplicated syntaxes and have been designed to reuse the code very easily through modules.

‍

ARM templates Vs Bicep

We all have worked with ARM templates and one of its main “challenges” are depending on the resource you deploy you will need to know exactly what to configure and, this can be quite frustrating. With Bicep, the syntax is very concise. You just declare the resource and that’s that.

It’s kind of you going to a pizza shop and mentioning what’s the pizza you would like to have, the size of it, and the toppings. You don’t tell them how to make the pizza dough, or how to pick the jalapenos or which supermarket you want the meat from.

See the below Bicep template which will declare a consumption based app service.

‍

The same app service plan done using ARM template will look like below

‍

The Bicep template is very straight forward, where as the ARM template syntax you need to be very explicit about what you require.

‍

Bicep playground

A good starting point will be the Bicep playground. In there you can experiment with Bicep. The most important feature I like in there is that the ability to transform your existing ARM templates into Bicep . You can simply click the “Decompile” button and point to the ARM template. But be mindful that all your ARM templates might not be easily converted into Bicep templates. You just need to fix the errors which it will show you so generously.

Required tools

• VSCode
• Bicep extension for VSCode

‍

‍

Absolutely love the Bicep extension! It has code snippets, syntax highlighting and, even intellisense!

You will find these features highly useful when building Bicep templates.

Resources required to deploy the function app

In here we are planning to deploy an Azure function app. First we’ll identify what are the resources required and create them using Bicep templates. Then like LEGO we’ll use these small building blocks to create the final template which we can use to deploy.

We will be using YAML based multi stage Azure DevOps pipeline to create the build and deployment pipeline.

To deploy an Azure function app you will need the below resources.

• Resource group
• Storage account
• App service plan
• App insights (optional but highly recommended)
• Key Vault (optional)
• Function App and its settings

Storage account

Let’s start with the storage account.

‍

The storage account is so straight forward. You get the storage account kind and the storage account tier as parameters and use them to declare the storage account you require. Finally in the output section it outputs the connection string to the storage account.

Application service plan

Every function app needs to be hosted. Most of the time this application service plan is already created and will be shared among other applications. But for this scenario I wanted to create a separate consumption based app service plan.

Application insights

Although this component is optional it’s highly recommended. The reason being you will be able to see your function app’s performance, failures in one central place.

Function app (without the settings)

First let’s create the skeleton function app without the settings. The key vault (next step) will need to know the function app’s principalid and the tenantid to provide access.

The function app which we’ll be creating will use deployment slots (azure function apps supports production and staging slots only).

So the below Bicep template creates function app with the two slots. Since the AKV will need the principalid and the tenantid the template will output them.

‍

Key vault

Although a key vault can be optional, in reality when developing applications most of the time we’ll have some settings which we would like to make them securely accessible. You can use an Azure key vault to securely save and version your secrets. The below template is accepting the function app’s principalid and the tenantid to setup the access policies.

Finally it outputs the URI (the latest version) of the secret to be used.

Function app settings

Finally let’s set up the configurations required for the function app.

Using these building blocks together

Now we have all the building blocks to create the function app. Bicep has this cool feature where you can create modules. Now lets use Bicep modules to organize our resources to be deployed.

‍

This is the storage account module

As you can see the syntax is very easy and straight forward. You define the module using the module keyword and the location for the Bicep template. Then you just simply pass the parameters required for the template. The Bicep extension of VSCode really helps you out here. As shown below the full intellisense is provided to you. It even give you a bunch of options including conditional access and my personal favourite required-properties .

But when deploying the Azure resources you will need to know the dependencies between them when deploying. In our function app these are the dependencies when the Azure resources need to be deployed.

• Function app has dependencies on the storage account and the application service plan.
• The key vault has a dependency on the function app because it needs to provide access to the function app.
• The function app settings module has dependencies on function app, app insights and the key vault module.
• The storage account, application service plan and the app insights modules does not have any dependencies.

The dependencies among the Bicep modules are specified as DependsOn. The function app module dependencies can be defined as shown below.

‍

Azure DevOps pipelines

Lets build a YAML based multi-stage pipeline to build and deploy our function app.

Build pipeline

The main purpose of a build pipeline is to check whether your code can be build successfully and to be able to create artifacts.

These are the steps associated with the build pipeline above,

• Build and restore your .NET projects.
• Run the tests.
• Create the function app artifact and package it as a zip.
• Create the Bicep templates as an artifact so that they can be referenced easily in the deployment pipeline.

Deployment pipeline

The steps involved in the deployment pipeline are as follows,

• Create the resource group.

We’ll use AZURE CLI to create the resource group.

Creating the resource group using Azure CLI

‍

The resource group is created only if it doesn’t exist.

‍

• Provision the resources

We have the main Bicep template which orchestrates all the required resources to deployed in Azure. You can use the same Azure CLI command to deploy resources, az deployment group create here.

Passing parameters to the main bicep template to provision resources

Notice the --template-file argument. Since we created all the Bicep templates as an artefact with the name deploy (see the build pipeline above) we can easily use it to locate the Bicep template. Then all the parameters which are required by the main.bicep template is passed.

‍

• Deploy to the staging slot (stop + deploy latest code + start)

Since we are deploying an HTTP triggered function app with slots, we need to deploy our code to the staging slot first. Let’s stop it first, secondly we’ll deploy the latest code there and then start the staging slot.

‍

• Deploy to the production slot (swap with staging + stop staging slot)

swap staging slot with production

‍

Since the staging slot is up and running, we can perform the swap operation with the production slot as specified above. Once it’s done no need to keep the staging slot alive. So as the final step we can stop the staging slot.

The build and deployment pipeline

Now we have a build pipeline and a deployment pipeline. In reality the applications we develop will be deployed in multiple environments. So let’s create the final piece where we can build and deploy the function app into multiple environments.

• Trigger points

We’ll trigger the pipeline when the code is checked in to the master branch or to any branch under feature . Also for PR requests made comparing the code to the master branch (you can make these conditional for each environment as well).

As you can see above there are multiple stages in the pipeline,

BUILD -> DEV -> SITare the stages in this pipeline, but you can create other environments as stages in the pipeline as you wish. Notice how the variable files are placed. There’s one common.yaml file to store all the parameters and each [env].yaml file to override the parameter values which will be specific to the environment.

Setting up the pipeline in Azure DevOps

Create a new pipeline in Azure DevOps, but since you already have the files required please select the file which has instructions to build and deploy your solution before you finish creating the pipeline in Azure DevOps.

Select the existing pipeline file in your solution.

‍

After running the pipeline, you will be able to see the resources successfully deployed in Azure.

The Azure DevOps pipeline

The created resource groups in Azure

The created resource groups

The deployment histories in resource groups.

‍

Finally, the resources deployed in the resource group.

The resources deployed in the resource group

Conclusion

Azure Bicep is awesome! The feature which I like most is the module support. VSCode and the Bicep extension is really helpful when you are building templates.

The code for this can be found in our GitHub

References

https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/overview

Integrating Security Controls Within a DevOps Pipeline

 Ref: https://www.mnemonic.no/globalassets/security-report/integrating-security-controls-within-a-devops-pipeline.pdf

	Tasks	Tool	More ..
Source	Apply security coding standards Compliance requirements  Configure risk thresholds Code review	Cics controls Mozilla risk assessment and Mirosoft threat modeling  Atlassian Crucible and GitLab	Paved road Rapid risk Assessment Code reviews
Build	Detect bugs or security issues in source code. dependency Analysis /Software Components Scanners are usually not able to detect flaws in business logic so perform unit test .	SAST: Checkmarx and Semmle.  Bandit (Python) and gosec (Go)	Static application static testing Dependency Analysis/software components Security Unit tests( high risk code)
Stage	Dynamic Application Security Testing (DAST) Scans for availability and integrity of data o pentest: discover flaws in architecture, configuration, and business logic Security attacks Create attack scenarios	Burp Suite Gauntlt, utilising external tools such as sslyze, nmap, and sqlmap	Dynamic application security test  Penetration testing  Automated security attacks
Production	create audit trails  protect the application runtime use centralised logging of all data sources, Detect security incidents securely store and access secrets	Sysdig/Falco, or Palo Alto Prisma	Run time protection Secret management Continuous security monitoring

Improving GraphQL performance in a high-traffic website

 Improving GraphQL performance in a high-traffic website

Best practices for optimising the responses to HTTP requests have long been established. We apply minification, compression, and caching as a matter of course. This raises the question: if this much effort is put into optimising the response, can we do more to optimise the request as well?

On realestate.com.au, we support client-side transitions between our search results pages and property detail pages. This involves sending an XHR request from the browser to a GraphQL server to retrieve the necessary data to populate the next page.

We recently applied an optimisation pattern called Automatic Persisted Queries (APQ) to these XHR requests which has reduced the median duration by 13%. We would like to use this post to share what we learned.

About GraphQL

To understand the problem APQ addresses, it helps to look at how GraphQL works. In a GraphQL API, the client sends a query that specifies exactly what data it needs. Each component in our React web application defines its data requirements in a GraphQL fragment and is assured that it will receive this data. Developers are given the flexibility to modify these fragments whenever data requirements change. This modularity is important for us because we have multiple squads of people contributing to the same React application.

Query:

{
  listing {
    address
    features   
  } 
}

Response:

{
  "listing": {
    "address": "511 Church St, Richmond",
    "features": ["community café", "sky bridge"]
  }
}

Applications like realestate.com.au require a lot of data points, for example, the property detail page requires data about the property, price, real estate agency, inspection times, market insights, etc. Having to specify every required field means that queries can become very large (22 KB for the property detail page). Large queries are not a problem for server-side requests that stay within Amazon infrastructure, but they do impact performance for client-side requests where consumers’ browsers are sending these queries in the payloads of XHR requests to the GraphQL server.

We considered a few approaches before landing on APQ as our preferred solution to this problem.

Approach 1: stripping ignored characters from the query

Due to the default settings in Apollo Client, we were sending GraphQL queries containing white-space and other decorative characters that made them human readable but inflated the payload by about 35%. Stripping out these machine-ignored characters is equivalent to the minification that we apply to HTTP responses. This approach was low development effort, because it only required changes to the React application, not the GraphQL server. Based on a cost benefit analysis we decided to implement this approach first before attempting APQ. The result was a 3.9% improvement to the median request duration.

Approach 2: de-duplicating fragments in the query

In our React application, each component defines its own GraphQL fragment. This makes the components modular and supports independent delivery by different squads. But it means there is some repetition in the query when multiple components request the same field. Some members of our team wrote a GraphQL fragment deduplication algorithm during REAio hack days to solve this problem. Deduping is similar to the compression we apply to HTTP responses, and would have further reduced the payload size. But we decided not to proceed with this approach due to it having a smaller benefit than approach 3.

Approach 3: persisting queries

If approach 1 is the minification, and approach 2 is the compression, then approach 3 is the caching of GraphQL queries. Instead of millions of users sending the same query string to the GraphQL server, this approach is for the server to cache a query after seeing it once, and then for all future requests to refer to that query with a hash. This approach effectively replaces a 22 KB query with a 64 byte hash. This approach was a higher cost because it required development in the React application and the GraphQL server, but after recording the improvement from approach 1 we decided this was a worthwhile investment.

How does it work?

Query hashes are fed into the GraphQL server’s persisted query cache at run-time. There is no need for build-time synchronisation between the React application and the GraphQL server. Here is the process:

Client generates a SHA 256 hash of the query.
Client sends an optimistic request to the GraphQL server, including a hash instead of a query.
If the GraphQL server recognises the hash, it responds with the full response. End of process.
Otherwise, it returns a special response type called PersistedQueryNotFound.
Client sends another request to the GraphQL server, this time including both the hash and the query.
The GraphQL server stores the query in a persisted query cache using the hash as a key. It responds with a full response.

Clients must make a second round-trip when there is a cache miss, but the vast majority of requests are a cache hit and only require one round-trip. Variables are not included in the hashed query. This means the same hash can be used for all property detail pages because the listing ID is passed in as a variable.

New query path

Optimised path

How did we implement it?

Automatic Persisted Queries is a pattern implemented in GraphQL libraries like Apollo and Relay. We are already using Apollo Client in our React application, so we just had to enable the feature there. Our GraphQL server is built on Sangria, which does not offer APQ, so our team built a custom implementation that adheres to the interface used by Apollo.

We built the implementation in a backwards compatible manner to ensure that the GraphQL server still supports other systems that do not yet use APQ, like our iOS and Android apps. When we released APQ in our React application, the GraphQL server was ready and waiting for those requests.

We were careful to put safeguards in place to protect against cache poisoning. This occurs when an attacker anticipates future cache keys (hashed queries) and sends requests to save invalid queries against those cache keys. To prevent this from happening, the GraphQL server will validate any hashes it receives before saving a new query to the cache store. When the GraphQL server receives a new query and hash, it hashes the query to check that the hash provided by the client matches the server-generated hash.

Results and next steps

Implementing Automatic Persisted Queries in realestate.com.au has improved the median duration of Ajax requests from by 13%. But we are really excited about another opportunity that this has unlocked. Now that the requests have such a small payload, we will be able to use GET requests rather than POST, which lets us use CloudFront caching in front of the GraphQL server. We expect that this will further improve the median request duration, and reduce the load on the GraphQL server. We will let you know how it goes!

Things to Consider When You Build a GraphQL API with AWS AppSync

When building a serverless API layer in AWS (one that provides a custom grammar for your serverless resources), your choices include Amazon API Gateway (REST) and AWS AppSync (GraphQL). We’ve discussed the differences between REST and GraphQL in our first post of this series and explored REST APIs in our second post. This post will dive deeper into GraphQL implementation with AppSync.

Note that the GraphQL specification is focused on grammar and expected behavior, and is light on implementation details. Therefore, each GraphQL implementation approaches these details in a different way. While this blog post speaks to architectural principles, it will also discuss specific features AppSync for practical advice.

Schema vs. Resolver Complexity

All GraphQL APIs are defined by their schema. The schema contains operation definitions (Queries, Mutations, and Subscriptions), as well as data definitions (Data and Input Types). Since GraphQL tools provide introspection, your Schema also becomes your API documentation. So, as your Schema grows, it’s important that it’s consistent and adheres to best practices (such as the use of Input Types for mutations).

Clients will love using your API if they can do what they want with as little work as possible. A good rule of thumb is to put any necessary complexity in the resolver rather than in the client code. For example, if you know client applications will need “Book” information that includes the cover art and current sales ranking – all from different data sources – you can build a single data type that combines them:

In this case, the complexity associated with assembling that data should be handled in the resolver, rather than forcing the client code to make multiple calls and manipulate the returned data.

Mapping data storage to schema gets more complex when you are accessing legacy data services: internal APIs, external REST services, relational database SQL, and services with custom protocols or libraries. The other end of the spectrum is new development, where some architects argue they can map their entire schema to a single source. In practice, your mapping should consider the following:

• Should slower data sources have a caching layer?
• Do your most frequently used operations have low latency?
• How many layers (services) does a request touch?
• Can high latency requests be modeled asynchronously?

With AppSync, you have the option to use Pipeline Resolvers, which execute reusable functions within a resolver context. Each function in the pipeline can call one of the native resolver types.

Security

Public APIs (ones with an external endpoint) provide access to secured resources. The first line of defense is authorization – restricting who can call an operation in the GraphQL Schema. AppSync has four methods to authenticate clients:

• API keys: Since the API key does not reference an identity and is easily compromised, we recommend it be used for development only.
• IAMs: These are standard AWS credentials that are often used for server-side processes. A common example is assigning an execution role to a AWS Lambda function that makes calls to AppSync.
• OIDC Tokens: These are time-limited and suitable for external clients.
• Cognito User Pool Tokens: These provide the advantages of OIDC tokens, and also allow you to use the @auth transform for authorization.

Authorization in GraphQL is handled in the resolver logic, which allows field-level access to your data, depending on any criteria you can express in the resolver. This allows flexibility, but also introduces complexity in the form of code.

AppSync Resolvers use a Request/Response template pattern similar to API Gateway. Like API Gateway, the template language is Apache Velocity. In an AppSync resolver, you have the ability to examine the incoming authentication information (such as the IAM username) in the context variable. You can then compare that username against an owner field in the data being retrieved.

AppSync provides declarative security using the @auth and @model transforms. Transforms are annotations you add to your schema that are interpreted by the Amplify Toolchain. Using the @auth transform, you can apply different authentication types to different operations. AppSync will automatically generate resolver logic for DynamoDB, based on the data types in your schema. You can also define field-level permissions based on identity. Get detailed information.

Performance

To get a quantitative view of your API’s performance, you should enable field-level logging on your AppSync API. Doing so will automatically emit information into CloudWatch Logs. Then you can analyze AppSync performance with CloudWatch Logs Insights to identify performance bottlenecks and the root cause of operational problems, such as:

• Resolvers with the maximum latency
• The most (or least) frequently invoked resolvers
• Resolvers with the most errors

Remember, choice of resolver type has an impact on performance. When accessing your data sources, you should prefer a native resolver type such as Amazon DynamoDB or Amazon Elasticsearch using VTL templates. The HTTP resolver type is very efficient, but latency depends on the performance of the downstream service. Lambda resolvers provide flexibility, but have the performance characteristics of your application code.

AppSync also has another resolver type, the Local Resolver, which doesn’t interact with a data source. Instead, this resolver invokes a mutation operation that will result in a subscription message being sent. This is useful in use cases where AppSync is used as a message bus, or in cases where the data has been modified by an external source, and notifications must be sent without modifying the data a second time.

GraphQL Subscriptions

One of the reasons customers choose GraphQL is the power of Subscriptions. These are notifications that are sent immediately to clients when data has changed by a mutation. AppSync subscriptions are implemented using Websockets, and are directly tied to a mutation in the schema. The AppSync SDKs and AWS Amplify Library allow clients to subscribe to these real-time notifications.

AppSync Subscriptions have many uses outside standard API CRUD operations. They can be used for inter-client communication, such as a mobile or web chat application. Subscription notifications can also be used to provide asynchronous responses to long-running requests. The initial request returns quickly, while the full result can be sent via subscription when it’s complete (local resolvers are useful for this pattern).

Subscriptions will only be received if the client is currently running, connected to the server, and is subscribed to the target mutation. If you have a mobile client, you may want to augment these notifications with mobile or web push notifications.

Summary

The choice of using a GraphQL API brings many advantages, especially for client developers. While basic considerations of Security and Performance are important (as they are in any solution), GraphQL APIs require some thought and planning around Schema and Resolver organization, and managing their complexity.