How Amazon’s IoT platform controls things without servers

CATEGORIES

Blog

Amazon’s IoT platform is a framework for connecting smart devices to the cloud. It aims to make the basic processes of collecting data and controlling devices as simple as possible. AWS IoT is a fully managed service, which means the customer doesn’t have to worry about configuring servers or updating operating systems. The platform simply exposes a set of APIs and automatically scales from a single device to millions of devices.

I recently wrote an article (in Finnish) in my personal blog about using AWS IoT for home automation. AWS IoT is not exactly designed for this purpose, but if you are tech savvy enough, it can be used for it. The pricing is currently set at $5 per million messages, which lasts a long time when you’re only dealing with a couple of devices sending occasional messages.

The home automation experiment provides a convenient context for discussing the basic concepts of AWS IoT. In the next few sections, I will refer to the elements of a simple home system that detects human presence in rooms and turns on the lights if it happens at a certain time of the day. All the devices are connected to the Amazon cloud via public Internet.

Device Registration

The first step in most IoT projects is to register the devices (also called “things”) into a centrally managed database. AWS IoT provides this database for free and lets you add any number of devices in it. The registration is important because each device also gets its own SSL/TLS certificate and private key, which are used for authentication and encryption. The devices can only be connected to AWS IoT by using their certificates and private keys.

The AWS IoT device registry also works as a simple asset management database. It lets you attach attributes to devices and maintain information such as customer IDs. The device registry can later be queried based on these attribute values. For example, you can find all devices belonging to a specific customer ID. The attributes are optional, so they can just be ignored if they’re not needed.

In the home automation experiment, two devices were added to the registry: A wireless human presence detector and a Philips Hue light control bridge.

Data Collection

Almost any IoT scenario involves collecting device data. Amazon provides the AWS IoT Device SDK for connecting devices to the IoT platform. The SDK is typically used to develop a small application that runs on the device (or on a gateway connected to the device) and transmits data to the cloud.

There are two ways to deliver data to the AWS IoT platform. The first one is to send raw MQTT messages, which are usually small JSON objects. You can then setup AWS IoT rules to forward these messages to other Amazon cloud services for further processing. In the home automation scenario, a rule specifies that all messages received under the topic “presence-detected” should be forwarded to an Amazon Lambda microservice, which then decides what to do with the information.

The other way is to use Thing Shadows, which are built into the AWS IoT platform. Every registered device has a “shadow” which contains its latest reported state. The state is stored as a JSON document, which can contain 8 kilobytes worth of fields and values. This makes it easy and cost-effective to store the current state of any device in the cloud, without requiring an external database. For instance, a device equipped with a thermometer might regularly report its current state as a JSON object that looks like this: {“temperature”:22}.

Moreover, It’s important to understand that Thing Shadows cannot be used as a general-purpose database. You can only look up a single Thing Shadow at a time, and it will only contain the current state. Indeed, you will need a separate database if you want to analyze historical time series of data. However, keep in mind that Amazon offers a wide range of databases you can easily connect to AWS IoT, by forwarding Thing Shadow updates to services like DynamoDB or Kinesis. This seamless integration between all Amazon cloud services is one of the key advantages of AWS IoT.

Data Analysis and Decision Making

Since Amazon already offers a wide range of data analysis services, the AWS IoT platform itself doesn’t include any new tools for analyzing data. Existing analysis services include products like Redshift, Elastic MapReduce, Amazon Machine Learning and various others. Device data is typically collected into S3 buckets using Kinesis Firehose and then processed by these services.

Device data can also be forwarded to Amazon Lambda microservices for real-time decision making. A JavaScript function will be executed every time a data point is received. This is suitable for the home automation scenario, where a single IoT message is sent whenever presence is detected in a room. The JavaScript function considers various factors, such as the current time of day, and decides whether to turn the lights on.

In addition to existing solutions, Amazon has announced an upcoming product called Kinesis Analytics. It will enable real-time analytics of streaming IoT data, similar to Apache Storm. This means that data can be analyzed on-the-fly without storing it in a database. For instance, you could maintain a rolling average of values and react to it instead of individual data points.

Device Control

The AWS IoT platform can control devices in the same two ways it collects data. The first way is to send raw MQTT messages directly to devices. Devices will react to the messages when they receive them. The problem with this approach is that devices might sometimes have network or electricity issues, which may cause the loss of some control messages.

Thing Shadows provide a more reliable way to have devices enter a desired state. A Thing Shadow will remember the new desired state and keep retrying until the device has acknowledged it.

In the home automation scenario, when presence is detected, the desired state of a lamp is set to {“light”:true}. When the lamp receives this desired state, it turns on the light and reports its current state back to AWS IoT as {“light”:true}. Once the reported state is the same as the desired state, the Thing Shadow of the lamp is known to be in sync.

User Interfaces and Data Visualization

You may use the AWS IoT Console to manually control devices by modifying their desired state. The console will show the current state and update it on the screen as it changes. This is, of course, a very low-level way to control lighting since you need to log in as a cloud administrator and then manually edit the JSON documents.

Then again, a better way is to build a web application that integrates to AWS IoT and offers a friendly user interface for controlling things. AWS provides rich infrastructure options for developing integrated mobile and web applications. Amazon API Gateway and Lambda are typically used to build a backend API that lets applications access IoT data. The data itself may be stored in a database like DynamoDB or Postgres. The access can be limited to authenticated users only using Amazon Cognito or a custom IAM solution.

For data visualization purposes, Amazon has recently announced an upcoming product called Amazon QuickSight, which will integrate with other Amazon services and databases. There are also many third-party solutions available through the AWS Marketplace. If any of these options doesn’t fit the use case well, a custom solution can always be developed as part of a web application.

My Findings

AWS IoT is a fast and easy way to get started on the Internet of Things. All the scenarios discussed in this article are based on managed cloud services. This means that you never have to maintain your own servers or worry about scaling.

For small-scale projects the operating costs are negligible. For larger scale projects, the costs will depend on the amount and frequency of the data being transferred. There are no fixed monthly or hourly fees, which makes personal experimentation at home very convenient.

Blog

Starter for 10: Meet Jonna Iljin, Nordcloud’s Head of Design

When people start working with Nordcloud, they generally comment on 2 things. First, how friendly and knowledgeable everyone is. Second,...

Blog

Building better SaaS products with UX Writing (Part 3)

UX writers are not omniscient, and it’s best for them to resist the temptation to work in isolation, just as...

Blog

Building better SaaS products with UX Writing (Part 2)

The main purpose of UX writing is to ensure that the people who use any software have a positive experience.

Get in Touch

Let’s discuss how we can help with your cloud journey. Our experts are standing by to talk about your migration, modernisation, development and skills challenges.








    How to accelerate digital transformation with culture, APIs, and cloud

    CATEGORIES

    Blog

    Digital transformation is constantly changing the way businesses and consumers interact with each other. In the simplest scenario, digitalisation is implemented by mapping a “physical” information process to a digital counterpart, aiming to produce the service more quickly and at a lower cost to customers. However, the greatest opportunity for using digitalisation is in business innovations that could disrupt the market. This may lead to a significant competitive advantage and should be the main objective for all companies going digital.

    We will briefly explain how to unleash the potential of digital transformation for your business, enabling you to cut costs and lead times for your future innovation projects.

    Develop a culture of experimentation

    To begin harnessing the benefits of digital business opportunities, your company will need to set up an innovation process including idea gathering, validation and creation of prototypes. The customer should be involved with the innovation process to enable constant feedback and validation. Doing so leads to continuous refinement of the product and a better match with customer expectations. Since the input from the market may be constantly shifting the final objective of the project, agile methodologies of software development should be chosen over waterfall models.

    The ability to validate concepts, as well as dismiss those that do not resonate with your customers, will quickly maximize the number of ideas that can be experimented with, and therefore increase the probability of focusing on initiatives that are more likely to succeed. Using this method of working will put your business in a competitive position in relation to your competitors.

    An example of this culture can be taken from the gaming industry, where Supercell dismissed fourteen potential projects in the process of creating its four blockbuster games. Only a small proportion of the projects started were ultimately developed into a finished product and launched to the public.

    APIfy your business to accelerate innovation

    To accelerate digital innovation, ensure that your internal business applications and processes are available externally via an easy to use but secure API. Loose coupling of your internal business data with external customer-facing applications reduces costs and accelerates innovation by preventing the need to repeatedly connect the source application to your external application to update recent changes.

    Not only do APIs improve productivity for creating new applications, but they may also enable customer, community or partner-driven application development initiatives, allowing third parties to build innovative applications on top of your own data. These third-party applications can lead to improved sales of your core offering and better customer satisfaction with minimal investments. In addition to the improved sales of your core offering, APIs may open new business opportunities via monetization of the data provided by the APIs.

    In the context of APIs, remember that developers are your customers. For successful API adoption, optimal developer experience, i.e. capability to find, understand and utilize the API, is crucial.

    Focus on value-adding work with the cloud

    Modern cloud platforms offer infrastructure (as a service) but also building blocks (Platform as a Service) that enable developers to focus on the development of the actual service, whilst leaving the infrastructure work to the cloud provider.

    In addition to traditional IT services such as storage and databases, cloud platforms also provide a large portfolio of more sophisticated services, including Internet of Things, data processing, and analysis; all at the click of a button. Such platforms enable development of high-end data-intensive solutions, with minimal up-front investment or commitment, minimal development effort and minimal delays.

    Cloud platforms are the perfect environment in which to develop and maintain applications from concept to production, as they offer both the flexibility required in the prototyping phase and the scalability and durability required by production systems.

    Blog

    Starter for 10: Meet Jonna Iljin, Nordcloud’s Head of Design

    When people start working with Nordcloud, they generally comment on 2 things. First, how friendly and knowledgeable everyone is. Second,...

    Blog

    Building better SaaS products with UX Writing (Part 3)

    UX writers are not omniscient, and it’s best for them to resist the temptation to work in isolation, just as...

    Blog

    Building better SaaS products with UX Writing (Part 2)

    The main purpose of UX writing is to ensure that the people who use any software have a positive experience.

    Get in Touch

    Let’s discuss how we can help with your cloud journey. Our experts are standing by to talk about your migration, modernisation, development and skills challenges.








      ARCHITECTURE FOR THE 21ST CENTURY GENERATION – THIS TIME, RE:IMAGINED

      CATEGORIES

      Blog

      Much like the last 5 years at re:Invent, we were treated on the Thursday to a keynote by Werner Vogels, speaking at the MGM Grand Garden Arena. It’s a huge space and the production values that AWS brings to their keynotes (coupled with the 16,800 capacity) made for an electric start to the morning.

      Vogels started the keynote by reflecting on the keynotes he has delivered over the last 5 years. During his first ever keynote back in 2012, Vogels discussed 21st-century architecture. He provided 4 guiding commandments: Controllable, Resilient, Adaptive, and Data Driven. He returned to this theme by calling this particular keynote ’21st Century Architectures, re:Imagined’

       

      It was made clear from the start that, unlike previous years, there would be relatively few announcements. He was true to his word, and instead focussed on just a few key themes. Vogels took time to thank AWS’s customers, reflecting that in the beginning, they knew they had to be collaborative to succeed. They wanted to build a collection of ‘nimble’ tools which could be assembled to build what customers needed. AWS listen to customer feedback, launching services that are rock solid, then working with customers to set the roadmap and development priorities.

      AWS want to help you build services for the future, and a lot of the announcements this week are enabled by developments in technology that have come about in the last 2-3 years.

       

      Voice As A Control System

      One of the themes Vogels spoke about was IoT and allowing whole environments to become accessible. Every device has the ability to become an input or output device, but with so many out there, it’s good to consider how we interact with all of them and their systems. Vogels believes that digital interfaces of the future will be human-centric, and the things that we as humans use to communicate will become the inputs to systems. The first of these will be the voice as it’s the most natural and easiest interaction.

      Once you can use your voice to control systems, Vogel suggested people won’t look back, from surgeons operating theatre equipment, to simply controlling the lighting or heating in your house, it will unlock digital systems for everyone.

      To demonstrate this point, Vogels talked about the International Rice Research Institute who provide rice farmers advice on how much and which fertiliser to put on their crops based on their years of research. Consumption of this information was very low until they invested in a voice interface. Farmers can call, select from one of 27 dialects, and provide information on their land and crop conditions. They then use voice recognition and machine learning to read back to the farmer which fertilizer they need.

      This was building up to the announcement of Alexa Business, a service that ‘makes it easier for you to introduce Alexa to your organization, providing the tools you need to set-up and manage Alexa enabled devices, enroll users, and assign skills at scale’

       

      Ensure You Are Well Architected

      The next theme of the keynote was architecture. Typically, systems have three planes: Admin, Control, and Data. (Vogels suggested architecture that extensive was difficult to visualise on marketing slides!) The AWS Well Architected Framework was launched two years ago and has grown from a single document to five pillars across five documents with two ‘lenses’. It guides the user on how to architect for specific use cases, (currently HPC and Serverless). The framework is included in AWS certifications and AWS regularly run boot camps and ‘Well Architected Reviews’ for its customers.

       

      Dance Like No One Is Watching, Encrypt Like Everyone Is

      This particular section had a strong focus on security and availability. On security, Vogels recapped everything you need to ensure you are doing, from implementing a strong identity foundation to automating security best practices. The need to encrypt everything was also highlighted and security has become a problem for all. Developers are now seen to be the new security team and everything needs to be remembered. For example, ensuring the security of the CI/CD pipeline, as well as ensuring security within the pipeline.

      Development has also changed over time, meaning you need to be more security aware. It’s more collaborative, there are more languages, and more services and teams are combining. To help out, AWS have launched Cloud9a cloud-based IDE, including a code editor, debugger, and a terminal pre-packaged with essential tools (JavaScript, PHP, Python), to allow you to write, run and debug your code, so you don’t need to set-up your development environments to start new projects.

       

      Everything Will Fail. All The Time

      Availability, reliability, and resilience were discussed, from the basics, (hard dependencies reduce availability, redundant dependencies increase availability) to the best practices of Distributed Systems, through to deployment automation and testing. Nora Jones (Netflix), gave the example of using Chaos Engineering and how they do this at Netflix.

      Vogels highlighted that available systems cost more and therefore it becomes a business decision whether to easily run something in a single availability zone, but only achieve 99% of uptime. If you want to increase this you need to distribute your services across multiple availability zones or even regions. DynamoDB Global Tables, for example, help you to do this, becoming the ultimate tool in reliability design. Although this has little to do with AWS (and more to do with decisions made within organisations), AWS can make this much easier for you. This brings us nicely onto the final part of the keynote – letting AWS do the ‘heavy lifting’ through its managed services.

      Galls Law says, “A complex system that worked is invariably found to have evolved from a simple system that worked”. AWS allows you to keep your systems simple by providing nimble services which you can assemble to build what you need. If you run your own RDBMS, you have to take care of the control and data planes. If you run on AWS, AWS manages the control plane. AWS Managed Services are designed for AWS to control the complex and hard to manage moving parts. making it simpler for you. This was demonstrated by Abby Fuller speaking about containers on AWS, and how Amazon Fargate can help you to make your environment much more simple. AWS will continue to release managed services over the next year.

       

      Serverless

      Serverless was something that couldn’t possibly be missed out of this keynote, with it being the ultimate AWS Managed Service. There is no server management, has flexible scaling, high availability, and no idle capacity. Here are the final (Lambda) product announcements

      In addition, the AWS Serverless Application Repository was also announcedallowing users to discover collections of serverless apps and easily deploy these into your account in a few clicks. You can also publish your own apps to share with the community, allowing you to easily consume their 3rd party Lambda functions and apply them to your environments.

       

      If you would like to understand how Nordcloud can help you take advantage of AWS Managed Services, help discuss whether your environment is well architected for, or discuss any other of the releases made this week, please get in touch. 

       

      Blog

      Starter for 10: Meet Jonna Iljin, Nordcloud’s Head of Design

      When people start working with Nordcloud, they generally comment on 2 things. First, how friendly and knowledgeable everyone is. Second,...

      Blog

      Building better SaaS products with UX Writing (Part 3)

      UX writers are not omniscient, and it’s best for them to resist the temptation to work in isolation, just as...

      Blog

      Building better SaaS products with UX Writing (Part 2)

      The main purpose of UX writing is to ensure that the people who use any software have a positive experience.

      Get in Touch

      Let’s discuss how we can help with your cloud journey. Our experts are standing by to talk about your migration, modernisation, development and skills challenges.








        Market leaders always push the envelope

        CATEGORIES

        Blog

        In this blog post, I will be picking up on what my colleague Sandip discussed in his latest blog post, ‘Innovating by Making a Difference’. Based on that, I wanted to take the opportunity to talk about how Nordcloud Germany have managed to stay on top of the industry for the last year or two. It’s been about focussing on the right things at the right time. For example, we haven’t worked in the Private Cloud space, and we haven’t been involved in the SaaS world of productivity, collaboration or CRM. We have stayed focussed purely on leading Public Cloud platforms; AWS, Azure & Google to deliver full-stack consultancy and services.

        At Nordcloud, we’re able to keep our customers – not just ourselves – on top of the game, by understanding everything we can, identifying the most valuable for our customers and then adopting the latest services of each of the providers. These are, for example, services around containers, (Kubernetes for instance), and serverless (Lambda), and also the Internet of Things and Machine Learning. Our work with companies of all industries and sizes is the foundation of being able to filter the different technologies for what matters the most. In this sense, our customers are those who teach us how to help them best and we can then pick the best technologies to do just that.

        We were recently screened by the leading Cloud market analyst in Germany against how we deliver state of the art managed Cloud services. Check out CRISP’s perspective here (in German). 

        We’re proud to be recognised as a leading provider in the Cloud consulting and service industry, who stands out amongst a vast number of peers in the market. If there is one thing we have realised throughout the years – both as a company and as individuals – it’s that you shouldn’t stop innovating and questioning. To stay on top, it’s not enough to just do the basics well. You have to keep going forward and step beyond your comfort zone at all times. At the same time, you shouldn’t be running after each new hype, but picking your game wisely and then building up expertise and concepts around that area.

        Blog

        Starter for 10: Meet Jonna Iljin, Nordcloud’s Head of Design

        When people start working with Nordcloud, they generally comment on 2 things. First, how friendly and knowledgeable everyone is. Second,...

        Blog

        Building better SaaS products with UX Writing (Part 3)

        UX writers are not omniscient, and it’s best for them to resist the temptation to work in isolation, just as...

        Blog

        Building better SaaS products with UX Writing (Part 2)

        The main purpose of UX writing is to ensure that the people who use any software have a positive experience.

        Get in Touch

        Let’s discuss how we can help with your cloud journey. Our experts are standing by to talk about your migration, modernisation, development and skills challenges.








          Keeping up with the latest skills: AWS IoT, Polly, and Rekognition

          CATEGORIES

          Blog

          Recently, I secured a number of AWS IoT Buttons for our office to play with and wanted to try to see how easy they would be to set-up and use in various mock-up applications. In the spirit of playing around with the buttons and keeping up my technical skills related to the AWS platform, I decided to make a small proof-of-concept project around them by collecting some old Android devices I had lying around, and various bits and pieces of AWS services such as Image recognition.

          The concept I finally settled with is a remote surveillance camera solution which can be triggered remotely with the AWS IoT Button, and which performs simple image recognition labelling the image content in the form of gender, roughage, mood, and other parameters. The solution will update a “monitoring” website where the latest surveillance image will be shown and the recognised characteristics spoke out for the viewer, removing the need to read the monitor in detail.

          For building the actual solution I selected the following tools and technologies together with the AWS platform:

          • Android tablet – I like to repurpose and recycle old and unused items, so I decided to use a decommissioned tablet as the IoT device which will act as the camera module for the system. Android devices are, in my opinion, one of the best toys to have lying around for building solutions requiring mobile, IoT, or embedded components. The platform is quite easy to use and easy to write applications in.
          • NodeRed – Since I didn’t want to spend too much time configuring and setting up the IoT libraries and framework in the Android devices, I decided to use NodeRed as the solution providing the MQTT protocol support, as it provides easy to use programming tools for doing quick PoCs around IoT. Running NodeRed requires SSH-access to the device, which I established using Termux and associated modules or controlling the camera etc.
          • The AWS IoT Button – This was an obvious choice as it was one of the technology components I wanted to test and one that also made me start working with the project in the first place.

          As the main idea of the solution was to build something around the AWS IoT Button and see how easy it is to set-up and use, this meant using the AWS platform as the IoT “backend”. For the rest of the solution, (as I didn’t want to start maintaining or setting up servers myself) I decided to use as many platform services as possible in AWS. I ended up working with the following AWS services:

          AWS IoT

          Using the AWS IoT platform for the message brokering, connectivities, and overall management of the IoT solution.

          AWS IAM

          The requirement here was to configure the various access roles and rights for all the architectural components in a secure way.

          AWS S3

          Using two distinct S3 buckets. One for uploading the images taken by the camera, one for hosting the website for the “monitoring” purposes.

          AWS Lambda

          Lambda functions were used to perform the required calculations and actions in a “serverless”-fashion and to remove the need for maintaining infrastructure components.

          AWS Polly

          Text-to-speech service used for creating the audio-streams required by the solution.

          AWS Rekognition

          Image recognition service used for analysing, and labelling the images.

          AWS CloudWatch and logs

          Used for monitoring and debugging the solution during the project.

          AWS CloudFormation

          Used for creating the resources, functions, roles etc. in the solution.

          Python/Boto3

          I selected to use Python as the programming language as the Boto3 libraries provide easy APIs to utilise the AWS services. Python was used to write all the Lambda functions to perform the processing required by the overall solution.

          How everything was brought together

          After registering the AWS IoT button (which was easily done with the AWS Android app), and Android devices to AWS IoT framework and provisioning the security credentials for them, they were good to be used as part of the solution. The architectural idea was to press a button to trigger a Lambda function which will do a few checks on the “upload” S3 bucket, creating a temporary signed URL for the S3 bucket. It will then use the AWS IoT topic to notify the Android devices on the image capture trigger. The Android device would then take the picture of whatever is standing in front of the camera and upload it securely to the “upload” S3 bucket using the temporary upload URL provided via the MQTT message it received earlier.

          Whenever new images are uploaded to the S3 bucket, this will trigger another serverless action in the background. This Lambda-function will take the image and use AWS Rekognition for performing the image recognition on it. The recognised labels and objects will then be run through AWS Polly to create the required audio stream. After all the new content is created, the Lambda-function will upload the content to the other S3 bucket where the website is hosted to show and play the content for whoever is watching the “monitoring” website. The separation of the S3 buckets provides added security measures, (a DMZ of sorts) to safeguard the website for the potentially harmful content which could, in theory, be uploaded to the upload bucket if the temporary upload URL was somehow acquired by an attacker.

          The whole solution is secured by AWS IAM by providing the least amount of necessary privileges for all the components to perform their actions in the exact resources they are using.

          Enabling Cloudwatch monitoring and logging is a good choice for debugging the solution, at least during the development phase. This enabled me to catch unnecessary typing errors in the granular IAM policies in the Lambda function’s IAM Role during the set-up.

          My findings

          This was a rather quick and fun project to work with and provided some insight into using the AWS IoT Button and Android devices as part of the AWS IoT ecosystem. The devices themselves were rather easy to get registered and functioning in the set-up. Of course in a large-scale real-world environment the set-up, certification creation, and installation of the IoT devices would need to be automated as well to make it feasible. Incorporating small Lambda-functions with image recognition and text-to-speech was quite straightforward and worked as a good learning platform for the technologies.

          When applying the project to a customer situation, I would definitely improve it by adding image transcoding for different screen sizes, create a proper web-service with searchable UI and proper picture database/index etc. All in all, I can highly recommend playing around with the IoT framework, IoT button, and NodeRed in Android. Creating these kinds of small side-projects is the perfect platform for people in our business to continue improving our skills and know-how around the ever-expanding technology selection in modern IT environment.

          Nordcloud offers deep-dive workshop which will help to identify the opportunities that impact your business and help you shape data-driven solutions which will take your business to the next level, contact us for more information.

          Blog

          Starter for 10: Meet Jonna Iljin, Nordcloud’s Head of Design

          When people start working with Nordcloud, they generally comment on 2 things. First, how friendly and knowledgeable everyone is. Second,...

          Blog

          Building better SaaS products with UX Writing (Part 3)

          UX writers are not omniscient, and it’s best for them to resist the temptation to work in isolation, just as...

          Blog

          Building better SaaS products with UX Writing (Part 2)

          The main purpose of UX writing is to ensure that the people who use any software have a positive experience.

          Get in Touch

          Let’s discuss how we can help with your cloud journey. Our experts are standing by to talk about your migration, modernisation, development and skills challenges.