Wednesday, September 30, 2020

A step by step guide to build an Event Grid viewer using serverless SignalR and Azure Functions

 In a microservice architecture we might have several decoupled microservices driven by events (aka event-based architecture). At some point, a developer might want to see the current status of the system, just a big picture of what is going on. In a classic monolithic system it might be easy since we only have one big service to look at. In an event-based architecture we can use an event viewer, a simple tool for printing the different events that are occurring in our system in live time.

In this post you will find a step by step guide to build a real-time event viewer for Azure Event Grid based on a serverless SignalR service.

The technology

First, let's take a look to the different services and tools we are going to use in this guide:

  • Azure Event Grid: a service for managing events. It takes care of routing the incoming events to the specific subscription based on filters.

  • Azure Function: a serverless computing service.

  • Azure SignalR: a service for enabling real-time web functionality to applications.

So, how do all these pieces fit together?

  1. SignalR Client application (in this case an Angular based app) must connect to the SignalR service. It makes use of a negotiation API exposed in an Azure Function to get the information to connect to the SignalR service, the SignalR url and a corresponding token.

  2. Our Azure Function will be triggered by new events in the system through Event Grid. Using Functions output binding it will push each event into the serverless SignalR service.

  3. Client application will receive in real-time those events using WebSockets.

Solution

Now let's move on to explaining the whole process step by step in detail.

1. Creating the Azure Function App

To create the Azure Function App we will use the Azure Function Core tools.

npm i -g azure-functions-core-tools@3 --unsafe-perm true

In a fresh new folder we will create the Function App and the Function:

func init
func new -> select http-trigger -> choose a name

Now we have a fresh new Azure Function ready to be launched and tested:

func start
curl http://localhost:7071/api/CloudEventSubscription

2. Subscribing to Event Grid

Azure Event Grid supports two different event schemas:

In this guide, we are going to use Cloud Event Schema v1.0. If you want to use Event Grid Schema you will need to make some minor changes to the validation subscription logic.

Azure Functions have a built-in trigger for Event Grid. However, it only works with Event Grid Schema. Cloud Events subscriptions must be done using a http trigger and doing the subscription validation manually.

To receive events in our Function we need to update the HttpTrigger input binding to allow OPTIONS and POST requests. With CloudEvents schema, OPTIONS verb is used for validating the subscription and POST for receiving the events.

[HttpTrigger(AuthorizationLevel.Function, "options", "post", Route = null)] HttpRequest req

Then add the logic to validate the subscription.

/*
Handle EventGrid subscription validation for CloudEventSchema v1.0
Validation request contains a key in the Webhook-Request-Origin
header, that key must be set in the Webhook-Allowed-Origin response
header to prove to Event Grid that this endpoint is capable of handling CloudEvents events.
*/
if (HttpMethods.IsOptions(req.Method))
{
    if(req.Headers.TryGetValue("Webhook-Request-Origin", out var headerValues))
    {
        var originValue = headerValues.FirstOrDefault();
        if(!string.IsNullOrEmpty(originValue))
        {
            req.HttpContext.Response.Headers.Add("Webhook-Allowed-Origin", originValue);
            return new OkResult();
        }

        return new BadRequestObjectResult("Missing 'Webhook-Request-Origin' header");
    }
}

The Function is ready to be subscribed to Event Grid, but we don't have the Event Grid resource created in Azure yet.

We are going to create the Azure resources directly from the console using Azure CLI. Let's start creating a new resource group.

az group create --name event-grid-viewer-serverless --location northeurope

Now we can create the Event Grid Topic in that resource. To do that we need to make use of an Azure CLI extension.

az extension add -n eventgrid
az eventgrid topic create --resource-group event-grid-viewer-serverless \
   --name topic --location northeurope --input-schema cloudeventschemav1_0

To create the subscription we need to expose the Azure Function using ngrok or any other similar tool.

az eventgrid event-subscription create \ 
   --source-resource-id /subscriptions/<your subscription id>/resourceGroups/event-grid-viewer-serverless/providers/Microsoft.EventGrid/topics/topic \
   --name serverless-signalr-function \
   --endpoint <your public Azure Function url>/api/CloudEventSubscription \ 
   --endpoint-type webhook \
   --event-delivery-schema cloudeventschemav1_0

Then we can test our fresh new subscription sending an event through Event Grid. To make the command easier we can create a sample CloudEvent event in a JSON file.

{
   "specversion":"1.0",
   "type":"com.serverless.event",
   "source":"mysource/",
   "subject":"123",
   "id":"A234-1234-1234",
   "time":"2020-06-14T12:00:00Z",
   "datacontenttype":"application/json",
   "data":"{\"eventProperty\":\"eventValue\"}"
}

The event can be sent using curl but we need to specify the Topic URL and Key that can be retrieved directly from the command line.

// Get the Topic URL
az eventgrid topic show --name topic \
   -g event-grid-viewer-serverless \
   --query "endpoint" --output tsv

// Get the Topic key
az eventgrid topic key list --name topic \
   -g event-grid-viewer-serverless \
   --query "key1" --output tsv

// Send the event
curl --request POST \
   --header "Content-Type: application/cloudevents+json; charset=utf-8" \
   --header "aeg-sas-key: <Topic Key>" \
   --data @event.json <Topic URL>

3. Enabling real-time

As explained at the beginning of the post, we will use SignalR for enabling real-time in our web application. Since we don't have a back-end but a serverless Azure Function for handling the SignalR connection we will use SignalR in serverless mode. We can create it easily with Azure CLI.

az signalr create --name serverless-signalr \
   --resource-group event-grid-viewer-serverless \
   --sku Free_F1 --service-mode Serverless \
   --location northeurope

Then we must install the Azure Functions Binding for SignalR. It will be used to ease the interaction between the Function and SignalR.

func extensions install -p Microsoft.Azure.WebJobs.Extensions.SignalRService -v 1.0.0

Client applications need some credentials to connect to the SignalR service. We need a new negotiate Function that, by making use of the Azure Functions Binding for SignalR, will return the credentials information to connect to SignalR.

[FunctionName("negotiate")]
public static SignalRConnectionInfo GetSignalRInfo(
    [HttpTrigger(AuthorizationLevel.Anonymous, "post")] HttpRequest req,
    [SignalRConnectionInfo(HubName = "hubName")] SignalRConnectionInfo connectionInfo)
{
    return connectionInfo;
}

By default, these bindings expect the SignalR ConnectionString to be set in a specific value on the application settings. To enable it locally we must add a new property to the local.settings.json file.

{
    "IsEncrypted": false,
    "Values": {
        "AzureWebJobsStorage": "UseDevelopmentStorage=true",
        "FUNCTIONS_WORKER_RUNTIME": "dotnet",
        "AzureSignalRConnectionString": "<Your SignalR Connection String>"
    }
}

Then, we want to push the incoming events from Event Grid to SignalR. First, we need to update the original Function to add an output binding for SignalR.

[SignalR(HubName = "hubName")] IAsyncCollector<SignalRMessage> signalRMessages

And add the logic to push the event to the SignalR service.

if(HttpMethods.IsPost(req.Method))
{
    string @event = await new StreamReader(req.Body).ReadToEndAsync();
    await signalRMessages.AddAsync(new SignalRMessage
    {
        Target = "newEvent",
        Arguments = new[] { @event }
    });
}

4. Client application

The last piece of the puzzle is the client application. First, it will negotiate the credentials with the Function, and then it will wait for new incoming messages from SignalR. In this tutorial we are going to use Angular (and Angular CLI) to create the client application.

ng new viewer-app

There is a npm package @aspnet/signalr that makes the use of SignalR in the client side very easy.

npm i @aspnet/signalr --save

This package handles for you the negotiation step and the connection to SignalR. We can add some simple logic to receive the events from a specific hub and log it into the console.

import * as SignalR from '@aspnet/signalr';

export class AppComponent {

  title = 'viewer-app';

  private hubConnection: SignalR.HubConnection;

  constructor() {
    // Create connection
    this.hubConnection = new SignalR.HubConnectionBuilder()
      .withUrl("http://localhost:7071/api")
      .build();

    // Start connection. This will call the negotiate endpoint
    this.hubConnection
      .start();

    // Handle incoming events for the specific target
    this.hubConnection.on("newEvent", (event) => {
      console.log(event)
    });
  }
}

To wrap up

In this tutorial we have created a simple but effective real-time event viewer for Event Grid using some cool stuff like serverless SignalR. This client application can be easily enhanced to show the events in the page rather than the console.

You can find a full example working with some minor UI enhancements on the following repo: https://github.com/DavidGSola/serverless-eventgrid-viewer

Azure Event Grid Viewer with ASP.NET Core and SignalR

 Important:  Code works without any errors.  Prety generic code . Build and publish to azure and use end points for any subscription in event grids . 

This blog post demonstrates how to create a site using ASP.NET Core and SignalR for the purpose of viewing notifications from Azure Event Grid in near-real time.

All the source code, along with the ARM template, can be found at https://github.com/Azure-Samples/azure-event-grid-viewer.

Introduction

Recently, I came across this great article from Andrew Stanton-Nurse on how to get started with SignalR on ASP.NET Core 2.1.0. Around the same time, a colleague of mine was searching for an alternative to RequestBin to review notifications from Azure Event Grid. While these are two very different topics, they somehow came together when working on a current project.

The primary goal was to display notifications from Event Grid in near-real time. Using ASP.NET Core and SignalR, the site should refresh as messages are received, providing the user with the experience of a real-time feed. In addition, the payload of each event should be available for inspection, just like RequestBin.

The screenshot below shows the results of the experiment:

viewer

Let’s walk through how the site was built and the lessons learned along the way.

Setup

To get started, I’ll need the following:

  1. The latest .NET Core SDK (at this time, it’s 2.1.300-rc1).
  2. npm – to download the SignalR Javascript library.

I’ll be using VS Code as an editor but that isn’t a requirement. The most important step is that the SDK is installed. To confirm the installation and other related information, we can execute the following command:

dotnet --info

This will return the version, along with additional information about the SDKs installed:

dotnet-info

The client-side library is something that will be added later, for now we will proceed with creating the site.

Creating the ASP.NET Core Site

Let’s start by creating a new site. For this project, I am going with the MVC template:

dotnet new mvc --name viewer

We’ll use a Web API controller to subscribe to messages from Event Grid – that’s why the MVC template was chosen. More on that later.

Configuring SignalR on the Server

The next step is to create a class in our project for the SignalR hub which will be used to push messages to the connected clients. Since this will be a one-way broadcast (from the server to the clients), the class is empty. If we wanted bi-direction communication between the clients and the server, then we would provide methods here as well.

using Microsoft.AspNetCore.SignalR;
namespace viewer.Hubs
{
public class GridEventsHub: Hub
{
public GridEventsHub()
{
}
}
}

Next, in the Startup.cs file, we need to add SignalR support. This begins by registering the service with the collection so that it can become available via dependency injection throughout the project:

// This method gets called by the runtime. Use this method to add services to the container.
public void ConfigureServices(IServiceCollection services)
{
services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_1);
// Awwww yeah!
services.AddSignalR();
}

Lastly, within the same Startup class, we update the Configure method to include a route to the hub we just created (lines 6-9). This basically provides a route for the clients to connect to and ultimately, receive messages from the server.

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
// other configuration …
// Add SignalR hub routes
app.UseSignalR(routes =>
{
routes.MapHub<GridEventsHub>("/hubs/gridevents");
});
app.UseMvc(routes =>
{
routes.MapRoute(
name: "default",
template: "{controller=Home}/{action=Index}/{id?}");
});
}

Receiving Events on the Server

With the basic configuration in place on the server-side, we can now move our attention to handling incoming notifications from Event Grid. Let’s create a strongly-typed class that represents the structure of an event. We’ll designate the Data property as generic:

using System;
namespace viewer.Models
{
public class GridEvent<T> where T: class
{
public string Id { get; set;}
public string EventType { get; set;}
public string Subject {get; set;}
public DateTime EventTime { get; set; }
public T Data { get; set; }
public string Topic { get; set; }
}
}

view raw
GridEvent.cs
hosted with ❤ by GitHub

Next, we create the API controller that will serve as the endpoint for incoming notifications from Event Grid.

Under the Controllers folder, let’s create a new class that inherits from Controller called UpdatesController. We’ll update the constructor to take in an instance of the hub and save it into a private data member:

using System;
using System.Collections.Generic;
using System.Linq;
using System.IO;
using System.Threading.Tasks;
using System.Net;
using System.Text;
using System.Net.Http;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using Microsoft.AspNetCore.SignalR;
using Microsoft.AspNetCore.Mvc;
using viewer.Hubs;
using viewer.Models;
namespace viewer.Controllers
{
[Route("api/[controller]")]
public class UpdatesController : Controller
{
private IHubContext<GridEventsHub> HubContext;
public UpdatesController(IHubContext<GridEventsHub> gridEventsHubContext)
{
this.HubContext = gridEventsHubContext;
}
}
}

We’ll also add two more data members that will inspect some header values to determine if the request is either a notification or request to validate a subscription from Event Grid:

private bool EventTypeSubcriptionValidation
=> HttpContext.Request.Headers["aeg-event-type"].FirstOrDefault() ==
"SubscriptionValidation";
private bool EventTypeNotification
=> HttpContext.Request.Headers["aeg-event-type"].FirstOrDefault() ==
"Notification";

The last piece of the puzzle for the controller is to handle the post request that will be called by Event Grid. Within this method we will:

  1. Handle any validation requests by returning the validation code (lines 60-64).
  2. Iterate through each event and broadcast it to all the connected clients (lines 68-81).

The implementation for the entire class is as follows:

using System;
using System.Collections.Generic;
using System.Linq;
using System.IO;
using System.Threading.Tasks;
using System.Net;
using System.Text;
using System.Net.Http;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using Microsoft.AspNetCore.SignalR;
using Microsoft.AspNetCore.Mvc;
using viewer.Hubs;
using viewer.Models;
namespace viewer.Controllers
{
[Route("api/[controller]")]
public class UpdatesController : Controller
{
private bool EventTypeSubcriptionValidation
=> HttpContext.Request.Headers["aeg-event-type"].FirstOrDefault() ==
"SubscriptionValidation";
private bool EventTypeNotification
=> HttpContext.Request.Headers["aeg-event-type"].FirstOrDefault() ==
"Notification";
private IHubContext<GridEventsHub> HubContext;
public UpdatesController(IHubContext<GridEventsHub> gridEventsHubContext)
{
this.HubContext = gridEventsHubContext;
}
[HttpPost]
public async Task<IActionResult> Post()
{
using (var reader = new StreamReader(Request.Body, Encoding.UTF8))
{
var jsonContent = await reader.ReadToEndAsync();
// Check the event type.
// Return the validation code if it's
// a subscription validation request.
if (EventTypeSubcriptionValidation)
{
var gridEvent =
JsonConvert.DeserializeObject<List<GridEvent<Dictionary<string, string>>>>(jsonContent)
.First();
await this.HubContext.Clients.All.SendAsync(
"gridupdate",
gridEvent.Id,
gridEvent.EventType,
gridEvent.Subject,
gridEvent.EventTime.ToLongTimeString(),
jsonContent.ToString());
// Retrieve the validation code and echo back.
var validationCode = gridEvent.Data["validationCode"];
return new JsonResult(new{
validationResponse = validationCode
});
}
else if (EventTypeNotification)
{
var events = JArray.Parse(jsonContent);
foreach (var e in events)
{
// Invoke a method on the clients for
// an event grid notiification.
var details = JsonConvert.DeserializeObject<GridEvent<dynamic>>(e.ToString());
await this.HubContext.Clients.All.SendAsync(
"gridupdate",
details.Id,
details.EventType,
details.Subject,
details.EventTime.ToLongTimeString(),
e.ToString());
}
return Ok();
}
else
{
return BadRequest();
}
}
}
}
}

Perhaps the most interesting piece of the notification component is the portion that sends the messages to the connected clients. The first parameter is the name of the method to invoke on the clients: gridupdate. The other parameters are a collection of properties we want to include in the message:

var details = JsonConvert.DeserializeObject<GridEvent<dynamic>>(e.ToString());
await this.HubContext.Clients.All.SendAsync(
"gridupdate",
details.Id,
details.EventType,
details.Subject,
details.EventTime.ToLongTimeString(),
e.ToString());

Implementing the Client

With the server portion complete, let’s move to the UI and retrieve the JavaScript library for SignalR using NPM:

npm install @aspnet/signalr

We’ll copy the signalr.js file (or it’s minified version – signalr.min.js) under the wwwroot/lib folder of the project.

The markup will consist of a table to display the events (lines 1-8) and a Handlebars template (lines 10-28) for binding the data from the message to the UI:

<table id="grid-events" class="table table-striped">
<thead>
<th>&nbsp;</th>
<th>Event Type</th>
<th>Subject</th>
</thead>
<tbody id="grid-event-details"></tbody>
</table>
<script id="event-template" type="text/x-handlebars-template">
<tr data-toggle="collapse" data-target="#event-{{gridEventId}}" class="accordian-toggle">
<td>
<button class="btn btn-primary btn-xs">
<span class="glyphicon glyphicon-eye-open"></span>
</button>
</td>
<td>{{gridEventType}}</td>
<td>{{gridEventSubject}}</td>
</tr>
<tr class="hiddenRow collapse" id="event-{{gridEventId}}">
<td colspan="12">
<div class="accordian-body">
<pre><code class="nohighlight">{{gridEvent}}</code></pre>
</div>
</td>
</tr>
</script>

Now to the part that bring this page to life: the JavaScript. Here is a breakdown of the remaining code:

  1. Include the script references for signalr, handlebars and a library that will help us display some JSON in a readable format (lines 2-6).
  2. Initialize the page by establishing a hub connection (lines 39-42).
  3. Handle incoming messages sent to ‘gridupdate’ (lines 45-47).
  4. Bind the data from the message to the UI using the handlesbars template (lines 17-30).

Note: jQuery, Bootstrap and other references are part of the generated shared layout (_Layout.cshtml) file that encapsulates this view.

@section scripts {
<script src="~/lib/signalr.min.js" type="text/javascript"></script>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css&quot;>
<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/highlight.min.js&quot;></script>
<script>hljs.initHighlightingOnLoad();</script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/handlebars.js/2.0.0/handlebars.js&quot;></script>
<script type="text/javascript">
var hubConnection;
var clear = function () {
$("#grid-events").find("tr:gt(0)").remove();
$("#grid-events").hide();
}
var addEvent = function (id, eventType, subject, eventTime, data) {
var context = {
gridEventType: eventType,
gridEventSubject: subject,
gridEventId: id,
gridEvent: data
};
var source = document.getElementById('event-template').innerHTML;
var template = Handlebars.compile(source);
var html = template(context);
$("#grid-events").show();
$('#grid-event-details').prepend(html);
}
var initialize = function () {
$("#grid-events").hide();
var clearEvents = document.getElementById('clear-events');
clearEvents.addEventListener('click', function () {
clear();
});
hubConnection = new signalR.HubConnectionBuilder()
.withUrl("hubs/gridevents")
.configureLogging(signalR.LogLevel.Information)
.build();
hubConnection.start().catch(err => console.error(err.toString()));
hubConnection.on('gridupdate', function (id, eventType, subject, eventTime, data) {
addEvent(id, eventType, subject, eventTime, data);
});
};
$(document).ready(function () {
initialize();
});
</script>
}

Our coding is done. Let’s build the project and publish it locally. To build:

 dotnet build

To publish the project to a local folder:

dotnet publish

We’ve written a good chuck of code to get this far. Let’s get this running on Azure!

Deploying to Azure

Prior to the RC1 release, we had to install the ASP.NET Core Runtime Extensions in order for this to run on an App Service – that is no longer a requirement. The only thing we need to configure on the server side will be to enable web sockets support from the application settings:

websockets

I’ve created a ARM template that will provision the App Service (with web sockets enabled) and deploy the code. You can kick it off from the GitHub repository: https://github.com/Azure-Samples/azure-event-grid-viewer.

Creating an Event Subscription

We’re in the final stretch – the site has been implemented and deployed onto an App Service in Azure. Our last step is to create an event subscription with Event Grid.

I happen to have a V2 storage account ready to test with but you can use any of the other event sources (even custom topics) for this exercise.

Under settings for the storage account, I will select Events:

storage-settings-eventgrid

From there, we can add a new event subscription:

new-eventsubscription

And then fill out the details for the subscription:

storage-eventsubscription

The subscriber endpoint will be something like:

https://<your-appservice-name>.azurewebsites.net/api/updates

The api/updates portion is for the API controller we created earlier. All the updates from Event Grid will be sent to that address.

End to End Testing

The animation below demonstrates the site in action. On the right are files being uploaded to a container. After each file is successfully uploaded, a notification is sent from Event Grid to the site. The left side, shows the site contents update dynamically for each incoming event.

viewer-animation-web

Summary

This was a fun project to put together. I always enjoy mashing up a bunch of technologies to create something new and unique – hopefully that is the case here.

Some key resources: