DynamoDB with Python and JavaScript

Amazon Web Services (AWS) is a probably the best known Cloud services offering on the Internet. At first glance AWS is a quite intimidating, learning about all the different features of AWS could be a full time job. For this blog I wanted to introduce one component of AWS called DynamoDB.

DynamoDB is a NoSQL database that could be connected to sensor hub devices such as Raspberry Pi’s for Internet of Things (IoT) solutions.

Getting Started with DynamoDB

The first step to getting started with AWS is to create an account. You can open a free “testing” account but unfortunately you’ll need to show that you’re serious by including a credit card number.

After creating an account you can sign into the AWS Management Console (https://aws.amazon.com/console/). With a single user account it is possible to run different services in different locations. It is important to note that not all services are available in all locations, so select a location that offers all the services that you need.

After a region is selected go to the Services menu item and search for DynamoDB, next select “create table”. A DynamoDB table has a NoSQL database structure, so this means that all that is required is a table name and a primary key. A primary key allows rows of data to have a way to be connected together.

For my IoT table example I created a primary key called signalname. (If you plan to have a very large data set and you need extremely high speed response you might want to consider defining multiple keys, such as location or sensor type).

Once the table is created, you can view and modify data.

The “Items” tab is used to add, view and delete data. Note, the viewing of updates is not automatic, so you need to press the “refresh” button periodically.

Users and Security

The AWS user security is quite detailed. To get started select your account name on the menu bar and “My Security Credentials“. The Groups item allows you to create security classifications for activities such as admin, viewing, etc.

New users can be defined and added into security groups.

When a new users is created an Access Key ID and a Secret Access Key is generated. The Secret Access Key is only shown when the user is created so ensure that you save it, and/or download the CSV file.

After an Access key ID and Secret Access Key is created for a user, you can start programming interfaces to DynamoDB.

DynamoDB and Python

The Python AWS library is called boto3 and it is installed in Windows by :

pip install boto3

and in Linux by:

sudo pip install boto3

The AWS security can be either added directly into the Python (or JavaScript) application or it can reside on the node or PC that you are working on. Both methods have pro’s and con’s. To add the credentials locally, you’ll need to install the AWS command line tool (awscli) and then enter your specific information:

$ sudo pip install awscli
$ aws configure
AWS Access Key ID [None]: AKIAIDxxxxx
AWS Secret Access Key [None]: kzzeko8Fxxxxxxxxxxxxxxxxx
Default region name [None]: Central
Default output format [None]:

A Python example to write and read a row into DynamoDB with the credentials in the program could be:

import boto3
import time

thesec = int(time.time() % 60)

mysession = boto3.session.Session(
    aws_access_key_id= 'AKIAID7xxxxxxxxx',
    aws_secret_access_key='kzzeko8xxxxxxxxxxxxxxxx')
db = mysession.resource('dynamodb', region_name='ca-central-1')

table = db.Table('my_iot_burl1')

response = table.put_item(
   Item={
        'signal': 'mytag1',
        'tagdesc': 'test tag1',
        'tagvalue': thesec
    })

response = table.get_item(Key={'signal': 'mytag1'})
print(thesec)
print(response)
print(response['Item']['tagvalue'])

If the credentials are locally used then the code to write data would be:

import boto3

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('users01')

table.put_item(
   Item={
        'signal': 'mytag1',
        'tagdesc': 'test tag1',
        'tagvalue': 16
    }
)

The response comes back as JSON data. For this example the result would be:

16
{‘Item’: {‘tagvalue’: Decimal(’16’), ‘tagdesc’: ‘test tag1’, ‘signal’: ‘mytag1’}, ‘ResponseMetadata’: {‘RequestId’: ’66M7R4CVCACA4KNA24LM90KG0NVV4KQNSO5AEMVJF66Q9ASUAAJG’, ‘HTTPStatusCode’: 200, ‘HTTPHeaders’: {‘server’: ‘Server’, ‘date’: ‘Fri, 15 Feb 2019 16:29:48 GMT’, ‘content-type’: ‘application/x-amz-json-1.0’, ‘content-length’: ’84’, ‘connection’: ‘keep-alive’, ‘x-amzn-requestid’: ’66M7R4CVCACA4KNA24LM90KG0NVV4KQNSO5AEMVJF66Q9ASUAAJG’, ‘x-amz-crc32’: ‘220267538’}, ‘RetryAttempts’: 0}}
16

For large amounts of data being written to DynamoDB it is possible to use a batch writing function:

import boto3
import time

db = boto3.resource('dynamodb', region_name='ca-central-1')

table = db.Table('my_iot_burl1')

thesec = int(time.time() % 60)

with table.batch_writer() as batch:
    for i in range(1,11):
        batch.put_item(
            Item={
                'signal': 'mytag' + str(i),
                'tagdesc': 'test tag'+ str(i),
                'tagvalue': thesec,
                'status': 'NORMAL'
            }
        )

The AWS console can be used to verify that the data was written in.

Writing to DynamoDB will clear the existing row with a completely new set of data. If you only want to update a field in a row then the update_item call is used. In the example below only the sensor tagvalue is updated and all other fields are left unchanged.

import boto3

mysession = boto3.session.Session(
    aws_access_key_id= 'AKIAIDxxxxxxxxx',
    aws_secret_access_key='kzzekxxxxxxxxxxxxxxx'
)

db = mysession.resource('dynamodb', region_name='ca-central-1')

table = db.Table('my_iot_burl1')

# set the value for mytag1 to bet 29
table.update_item(
    Key={ 'signal': 'mytag1' },
    UpdateExpression='SET tagvalue = :newval1',
    ExpressionAttributeValues={
        ':newval1': 29
    }
)

Python “Scan the Data” Example

The boto3 library supports the ability to scan the data based on some filtering criteria. The filter syntax is: Attr(‘the field’).eq(‘thevalue’).

The following comparison operators are available:

eq | ne | le | lt | ge | gt | not_null | null | contains | not_contains | begins_with | in | between

Multiple statements can be created with | (OR) , & (AND) operators. Below is an example that scans the database for rows with an alarm status or with a tag value greater than 100.

import boto3
from boto3.dynamodb.conditions import Key, Attr

db = boto3.resource('dynamodb', region_name='ca-central-1')

table = db.Table('my_iot_burl1')

response = table.scan(
    FilterExpression=Attr('status').eq('ALARM') | Attr('tagvalue').gt(100)
)

#print (response)
for i in response['Items']:
    print(i)
    print(i['signal'], "=", i['tagvalue'])

The results were:

{‘tagvalue’: Decimal(‘150’), ‘signal’: ‘mytag2’, ‘tagdesc’: ‘test tag2’, ‘status’: ‘NORMAL’}
mytag2 = 150
{‘tagvalue’: Decimal(‘999’), ‘signal’: ‘mytag1’, ‘tagdesc’: ‘test tag1’, ‘status’: ‘ALARM‘}
mytag1 = 999

JavaScript

For standard HTML/JavaScript projects the credentials are included in the source file. This obviously has some serious security issues, so it’s recommended that you define a low level user that can only do a small subset of functions.

Below is a simple example that can read our DynamoDB table and it can write a new row with an updated value.

<html>
<head>
<script src="https://sdk.amazonaws.com/js/aws-sdk-2.7.16.min.js"></script>

<script>
AWS.config.update({
  region: "ca-central-1",
  accessKeyId: "AKIAID7xxxxxxxxxxxx",
  secretAccessKey: "kzzeko8xxxxxxxxxxxxx"
});

var docClient = new AWS.DynamoDB.DocumentClient();

function createItem() {
  var d = new Date();
  var n = d.getSeconds();
    var params = {
        TableName :"my_iot_burl1",
        Item:{
      'signal': 'mytag4',
      'desc': 'test tag4',
      'info': {
        'value':n,
        'quality': "questionable"
        }
    }
    };
    docClient.put(params, function(err, data) {
        if (err) {
            document.getElementById('textarea').innerHTML = "Unable to add item: " + "\n" + JSON.stringify(err, undefined, 2);
        } else {
            document.getElementById('textarea').innerHTML = "PutItem succeeded: " + "\n" + JSON.stringify(params,undefined, 2);
        }
    });
}

function readItem() {
    var table = "my_iot_burl1";
    var signal = 'mytag4';
    var params = {
        TableName: table,
        Key:{'signal' : 'mytag4' }
    };
    docClient.get(params, function(err, data) {
        if (err) {
            document.getElementById('textarea2').innerHTML = "Unable to read item: " + "\n" + JSON.stringify(err, undefined, 2);
        } else {
            document.getElementById('textarea2').innerHTML = "GetItem succeeded: " + "\n" + JSON.stringify(data, undefined, 2);
        }
    });
}

</script>
</head>

<body>
<input id="createItem" type="button" value="Create Item" onclick="createItem();" />
<br><br>
<textarea readonly id= "textarea" style="width:400px; height:300px"></textarea>
<hr>
<input id="readItem" type="button" value="Read Item" onclick="readItem();" />
<br><br>
<textarea readonly id= "textarea2" style="width:400px; height:300px"></textarea>
</body>
</html>

Summary

On first impressions I was pretty overwhelmed using Amazon Web Services. However after some time playing with it I found that it wasn’t unlike many other Web services that are available.

DynamoDB can be used to communicate with sensor values from a Raspberry Pi, but unfortunately there are no mainstream libraries for Arduino projects. AWS does offer an MQTT solution that can be used to update a DynamoDB table.

For small IoT projects I feel that AWS is overkill, but for larger projects I would definitely consider using it.