Choosing the right tools for the job

Best Practices for your Azure IoT Data Ingress Layer

Azure IoT reference architecture

Source: https://aka.ms/iotrefarchitecture

Azure IoT reference architecture

Source: https://aka.ms/iotrefarchitecture

Data Ingress

  • Cloud Gateway is a solved problem: Azure IoT Hub
  • Data Ingress is about moving data as fast as possible from the cloud gateway into stream processors & storage

Responsibilities of the data ingress layer

  • Deserializing arriving packages
  • Making security & plausibility checks
  • Routing data forward into the system

Data Ingress - Available Azure services

IoT Hub Data Routing

Routes define
  • What kind of data (device messages, lifecycle events, device twin changes)…
  • … with which properties (filtering)…
  • … goes where by selecting an Endpoint (=data sink, e.g. Event Hubs, Service Bus Queues & Topics, Blob Storage)

Data Ingress - Available Azure services

Stream Analytics

Data Ingress - Available Azure services

Stream Analytics

Inputs can be
  • Events streams from an IoT Hub or Event Hub (currently only JSON, CSV and AVRO serialization formats are supported)
  • Reference data for joining from a blob storage
Outputs can be
  • Almost anything
Routing is implemented via a SQL-like syntax
  • SELECT * INTO output FROM input ... WHERE, GROUP BY

Data Ingress - Available Azure services

Functions

Data Ingress - Available Azure services

Functions

Inputs / Triggers can be
  • HTTPS requests
  • Events streams from an IoT Hub or Event Hub, Messages from a Service Bus
  • Updated / Inserted data in a Blob or CosmosDB
Outputs can be
  • Literally anything that has an API
Routing is implemented with the programming language of your choice
  • C#, F#, JavaScript, Java (in preview)

Data Ingress - Available Azure services

General Compute Services - such as Service Fabric

Data Ingress - Available Azure services

General Compute Services - such as Service Fabric

Inputs / Triggers can be
  • Literally anything that has an API
Outputs can be
  • Literally anything that has an API
Routing is implemented with the programming language of your choice
  • For Service Fabric: all .NET languages

Comparing Functions & Service Fabric

Experiences from a load test in a real world project

3000 simulated devices sent a total of 1.200 messages per minute (which isn't very much in IoT scnearios) over several hours to an IoT Hub. Two different architectures were evaluated
There were no complex requirements regarding data aggregation or analyzing the event stream, but the messages were serialized using Protobuf, which ruled out Stream Analytics

Comparing Functions & Service Fabric

Experiences from a load test in a real world project

The Functions based approach turned out to be...
  • faster - processing a message from IoT Hub into the storage took only ~200ms (compared to ~500ms-700ms in the Service Fabric approach)
  • cheaper to run - using Service Fabric, even for development environments you'd need a cluster of several machines, while the Functions consumption plan could handle the production load nearly for free (sorry, Microsoft 🙂)
  • easier to develop - being able to use existing components such as the Function Event Hub bindings saved us a ton of boilerplate code

... faster, cheaper, easier - this sounds too good to be true, right?

Azure IoT Data Ingress cheat sheet

  • You just want to persist the data flowing into your IoT Hub, without transformation or other logic?
    → Use IoT Hubs built-in routing
  • Your data is serialized as JSON, CSV or AVRO, and the reference data you need to correlate an event stream to does not change (often)?
    → Consider using Stream Analytics. Depending on if you have a Stream Analytics job anyway or not, and also on your data throughput, a Function might be cheaper, though
  • Do you expect complex processing logic, a lot of caching issues, a very high load, or do you need a Service Fabric cluster anyway?
    → Use Service Fabric Actors and Event Processor Host
  • In many other cases...
    → Consider using a Function

Thanks for listening

Questions?

You can find the slides at
https://christianeder.github.io/azure-iot-slides/ingress-best-practices
And a more detailed view on the architecture at
https://github.com/christianeder/azure-iot-reference-architectures