Storage Account for Azure Stream Analytics Monitoring

Azure Stream Analytics (ASA) is a Platform-as-a-Service (PaaS) for consuming streaming data from different types of input. It primarily reads the data stream, processes it and then sends the data to the output. The input and output could be any other Azure services like IoT hub, event hubs, service bus queue and topic, blob storage, power BI etc. This service is recommended for IoT scenario to ingest the stream of data from the device to cloud (D2C) and form a hot and cold path analytics.

To check the throughput of the highly loaded IoT and transactional system, a performance load test is required to find the streaming unit(SU) % utilization and ascertain how ASA jobs can scale up under the load. By default, stream analytics comes with key metrics data like input events, output events, SU% utilization etc. and display them on the monitor tab of the Azure portal page. These metrics help us to check the performance of the ASA job. A screenshot is shown belowsnip_20151122124436
The metrics data can be used to monitor the stream analytics job. Under the hood, these metrics data is stored in the table storage. The name of the table storage is given at the time of creating the stream analytics job. When ASA job runs, it logs the metrics data in five different tables and the process happened internally by the execution engine. It is worthwhile to mention that there is a certain delay (like 5 to 10 minutes) to transfer the data to the table storage. The naming convention of the tables is given below

  • WADMETRICSP1DP10DV2S20150819
  • WADMETRICSPT30MP10DV2S20150819

It is important to check the naming rule because they follow certain patterns. The details are given below

  • P1D – Data is aggregated over 1 day
  • PT1H – Data is aggregated over 1 hour
  • PT1M – Data is aggregated over 1 minute
  • PT30M – Data is aggregated over 30 minute
  • PT6H – Data is aggregated over 6 hours

The date format is YYYYMMDD and the tables would be created at the interval of 10 days based on the start date. If your ASA job starts on 09-Jan-2015 then the name of the tables would be 20150109, 20150119, 20150129 and so on.
Once the data is stored, the next step is to consume the metrics data and lets see how can they be retrieved from the table storage.

This code helps to show the metrics data in tabular format to the console window. Then the data can be used by any data visualization tool (Power BI, D3.js etc) and show them in the chart to understand the behaviors of ASA jobs. Please remember to change the event type in line number 19,20,21 as per the need and change the metrics parameter to Average in case of SU % utilization in line number 61,62.