Introduction
The growth of public cloud providers has made the unthinkable thinkable. Iot, Machine Learning, AI etc all of these technologies would need enormous amounts of effort which includes building up the infrastructure, designing, choosing the right tools and technology etc, the list goes on. Here comes AWS cloud which solves a lot of complexities in standing up the infrastructure and IoT is one such area where AWS has tremendously evolved over the last few years.
This article discusses an IoT use case that we solved for a customer where they wanted to quickly on-board their IoT devices, track vulnerabilities, analyze the device data, notify the end-users of any patterns that do not adhere to their daily practice and fix issues before they blow out of proportion.
Problem Statement
Customer has developed an IoT device and is interested in providing services to its end customers. Customer wanted to have a connected infrastructure to be able to measure key statistics and trends, and provide data analytics. The customer has deployed the devices all over the continental US and had challenges determining the state of the device and analyzing the data.
As depicted below customer deployed devices were spread across all over the place and the customer really had no way to track the health and wellness of the device and provide analytical data to their clients.
Solution
AWS IoT Core and IoT Analytics can be leveraged to address this use case. The customer devices are spread across with no connectivity to be able to determine the status of the device. The customer device was programmed to connect to AWS cloud via MQTT protocol.
MQTT is a lightweight and widely adopted messaging protocol that is leveraged for connecting remote devices. It is an OASIS standard messaging protocol for the Internet of things (IoT).
Customer devices came built in with a module that had no connectivity. These modules were reprogrammed to leverage MQTT which gave the devices the ability to connect to AWS cloud. The MQTT protocol was then leveraged to connect to AWS IoT Core to further process and provide data analytics.
The solution involved in building an AWS IoT core service that sent raw data to an S3 bucket which was then re-processed to transform the data into a much readable format for analytics and then leverage AWS Athena to store and provide analytics via AWS Quicksight.
The steps detail the flow of events from the registration to ingesting a message in S3 and feeding it to Quicksight for analysis.
- Copy the device certificate and SDK (module specific to the device to be able to communicate via MQTT protocol) to the IoT Data Device
- IoT Device emits data events from the connected device.
- The device events use the MQTT protocol of AWS SDK to connect to AWS Cloud.
- The device connects to AWS IoT Core.
- The device messages/events are subscribed to an IoT topic.
- The message subscribed in the IoT device is then processed by an IoT Rule.
- The IoT rule puts the message into an S3 bucket.
- AWS API Gateway is subscribed to the S3 bucket to listen to the raw messages coming in from Iot Device.
- The message from API Gateway is processed by a Lambda function to transform the message into a flatter structure to be able to analyze data.
- The Lambda function transforms the message and puts the message into an S3 bucket.
- Aged data is then pushed to S3 glacier for reducing the cost.
- A crawler, AWS Glue is then run against the processed data to ingest the data into Athena.
- The data in S3 bucket is ingested to AWS Athena service for storing data in SQL like format via the AWS Glue.
- AWS quicksight can then be leveraged to point to the data in Athena service to provide visualization and analytics.
References
- AWS Iot Core – https://aws.amazon.com/iot-core/
- MQTT – https://mqtt.org/