Configure Azure Stream Analytics Alert Rule using Insights REST API

Azure Stream Analytics (ASA) helps us to quickly develop the real time streaming solution for Internet of Things (IoT) scenarios such as processing messages from the real devices. In an actual IoT implementation, there could be multiple ASA jobs to process different types of messages from the devices and perhaps this number could go beyond hundreds. The question is how the monitoring would happen for this large number of ASA jobs to make sure that there is no issue in message processing by the ASA job.

The answer is using automated alert for Azure Platform-as-a-Service (PaaS) out of the box feature. The alert rule can be created based on the available metrics with the threshold value and the mail address. The mail would be triggered by the PaaS service when the condition satisfies and action can be taken to resolve the issue by the engineers. Lets take a look how to setup an alert rule for an ASA job from the Ibiza portalI have already created an ASA job to process the message from IoT Hub to Blob Storage. The screenshot is given below
Once the ASA job is created and in running state, it starts capturing the metrics data like input events, output events, data conversion errors, runtime errors, SU% utilization etc as depicted below
snip_20160123192552The metric events can also be used to setup the alert which is triggered when the condition satisfies. The following alert rule triggers when SU% utilization is more than 90% for the duration of 15 minutes.
snip_20160123201543The above steps outline to set up the alert rule for an ASA job. If the number of ASA job is less than 5 or 10, the steps are achievable but if the number is significantly high (scale of hundreds) then it is a cumbersome process to create the alert rule for the ASA jobs. In order to create the alert rule programmatically, Insights REST API is used with the Azure PowerShell module and the process is really straightforward.The Insights REST API is authenticated using Azure Active Directory and the details can be found in my earlier post.  I will use a json file to store the name of the ASA jobs and details of the alert rules.

"RuleName":"asa error rule",
"RuleDescription":"This is asa error rule description",
"RuleName":"asa error rule one",
"RuleDescription":"This is asa error rule one description",

Following script is written using Azure Powershell module

#Path to Active Directory Authentication Library (ADAL) in the local machine.
#Get the ADAL dll by adding nugget package Microsoft.IdentityModel.Clients.ActiveDirectory and copy the dll from the packages folder
$adal = "D:Microsoft.IdentityModel.Clients.ActiveDirectory.dll"

#Load the ADAL Library

$collectionWithItems = @()

#Setting the configuration data
#Replace <Tenant Name>
$adTenant = "<tenantname>"
#Replace <User Name> and <Tenant Name>
$username = "<username>@<tenantname>"
#Replace <Password>
$password = "<password>"
$resourceAppIdURI = ""
#Replace <Tenant Name>
$authority = "<tenantname>"

#Connecting to AAD through ADAL
$credential = New-Object "Microsoft.IdentityModel.Clients.ActiveDirectory.UserCredential" -ArgumentList $username, $password
$authContext = New-Object "Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext" -ArgumentList $authority, false

#Acquire the access token
$authResult = $authContext.AcquireToken("", "4c177323-daac-4a45-a587-3baadb54650b", $credential)

Write-Host -ForegroundColor Green "Connection Successful with accesstoken - " $authResult.AccessToken

#Get the Content from the json file
$referencedata = Get-Content -Raw -Path "ReferenceData.json" | ConvertFrom-Json

foreach($JobDetails in $referencedata.JobDetails)
 #Replace <Tenant ID>
 $strUrl =" ID/resourceGroups/StreamAnalytics-Default-East-Asia/providers/microsoft.insights/alertRules/$JobDetails.RuleName?api-version=2014-04-01"
 #Replace <Tenant ID> and <Email ID>
 $jsonData = "{`"location`":`"East Asia`",`"tags`":{}, `"properties`": {`"name`": `"" + $JobDetails.RuleName +"`" , `"description`": `"" + $JobDetails.RuleDescription + "`", `"isEnabled`":true, `"condition`":{ `"odata.type`": `"Microsoft.Azure.Management.Insights.Models.ThresholdRuleCondition`", `"datasource`": {`"odata.type`": `"Microsoft.Azure.Management.Insights.Models.RuleMetricDataSource`", `"resourceUri`": `"/subscriptions/<Tenant ID>/resourceGroups/" + $JobDetails.ResourceGroupName + "/providers/Microsoft.StreamAnalytics/streamingjobs/"+ $JobDetails.ASAJobName + "`",`"metricName`":`"Errors`"},`"threshold`":`"1500`",`"windowSize`":`"PT15M`",`"timeAggregation`":`"Total`"},`"action`":{`"odata.type`":`"Microsoft.Azure.Management.Insights.Models.RuleEmailAction`",`"sendToServiceOwners`":`"true`",`"customEmails`":[`"<Email ID>`" ]}}}"

$item = New-Object System.Object

$item | Add-Member -MemberType NoteProperty -Name "ASAName" -Value $JobDetails.ASAJobName
$item | Add-Member -MemberType NoteProperty -Name "RuleName" -Value $JobDetails.RuleName
$item | Add-Member -MemberType NoteProperty -Name "Url" -Value $strUrl
$item | Add-Member -MemberType NoteProperty -Name "jsonData" -Value $jsonData

$collectionWithItems += $item

foreach($dataItem in $collectionWithItems)
 Invoke-WebRequest -Method PUT -Headers @{
 Authorization = $authResult.CreateAuthorizationHeader()
 'Content-Type' = "application/json"
} -Uri ($dataItem.Url) -Body $dataItem.jsonData

Write-Host -ForegroundColor Green "Alert Rule" $dataItem.RuleName "created sucessfully for" $dataItem.ASAJobNameName

The above powershell script does following jobs

  1. Authenticate over Azure Active Directory using username and password
  2. Get the JWT access token from Azure Active Directory
  3. Read the input from ReferenceData.json file
  4. Call the Insights REST API with the access token and input json data

In Step 4, the input json data is populated to post to the Insights REST API. The condition is on the total error metrics with threshold value of 1500 over a period of 15 minutes. The details of the json data is given below

"location":"East Asia",
"name":"<Name of the Alert Rule>",
"description":"<Description of the Alert Rule>",
"resourceUri":"/subscriptions/<AAD Tenant ID>/resourceGroups/<Resource Group Name>/providers/Microsoft.StreamAnalytics/streamingjobs/<ASA Job Name>",
"customEmails":"[Alert Recipient's Email ID]"

More details can be found here for the input json data. The alert rule would be in action once the script execution is completed and it does not require any restart of the ASA jobs.