RabbitMQ REST API

RabbitMQ is an open source middleware message solution that natively uses AMQP communications but it has a good selection of plug-ins to support features like: MQTT, MQTT Web Sockets, HTTP REST API and server-to-server communications.

rabbitmq_overview

RabbitMQ has a very good selection of interfacing clients for most of the major programming languages. For applications where there is no custom programming library the REST API can be used.

Applications where the REST API could be used might include:

  • Arduino IoT projects
  • Web Clients (not using NodeJS)
    • For simple reading and writing of messages
    • For monitoring of RabbitMQ loading
  • AppInventor Android apps

Unfortunately I didn’t find the RabbitMQ REST API to be 100% straightforward, so in this blog I could like to document some of my notes and testing.

What is REST API

Representational State Transfer (REST) is a software style for creating Web services. A client issues an HTTP request to a server with a GET/POST/PUT or DELETE method and the server responds with a JSON (Javascript Object Notation) formatted message.

RESTAPI_overview

The most common tool to test REST API is curl. Curl is supported on all major OS’s. The curl syntax will vary slightly between Windows and Linux, especially in how the systems handle strings.

RabbitMQ Setup

RabbitMQ can be installed on Window, Linux, MacOS systems and there are also some cloud based offerings. For small systems lower end hardware like a Raspberry Pi can be used.  For complete RabbitMQ installation instructions see: https://www.rabbitmq.com/download.html . To install and run RabbitMQ on a Ubuntu system enter:

sudo apt-get update
sudo apt-get install rabbitmq-server
sudo service rabbitmq-server start

The next step is to add some plug-ins. For the Web Administration and REST API plug-ins:

sudo rabbitmq-plugins enable rabbitmq-management

The rabbitmqctl command line tool allows you to configure and review the RabbitMQ server. To add a user pete,  with password pete, that has config, write and read rights for management and administrator access, enter:

sudo rabbitmqctl add_user pete pete
sudo rabbitmqctl set_permissions -p / pete ".*" ".*" ".*"
sudo rabbitmqctl set_user_tags pete management administrator

After you’ve defined an administrative user the RabbitMQ web management plug-in can be accessed by: http://ip_address:15672 .

Rabbit_webadmin

Once the Web Management component is installed it is much easier to test and play with REST API calls.

REST API Calls Using CURL

For a complete set of RabbitMQ REST API calls see: https://pulse.mozilla.org/api/ .

Typically you would use the RabbitMQ tools to create and exchanges and queues but it is possible to do this via the REST API interface.

Below are some examples using user: pete and a password of pete . Note: %2f is used for default (blank name) virtual host.

Create an exchange: webex

curl -i -u pete:pete -H “content-type:application/json” -X PUT http://192.168.0.116:15672/api/exchanges/%2f/webex -d”{“”type””:””direct””,””auto_delete””:false,””durable””:true,””internal””:false,””arguments””:{}}”

Create a Queue: webq1

curl -i -u pete:pete -H “content-type:application/json” -X PUT http://192.168.0.116:15672/api/queues/%2f/webq1 -d”{“”auto_delete””:false,””durable””:true,””arguments””:{}}”

Create Binding: Bind queue webq1 to exchange webex with a routing key of webq1

curl -i -u pete:pete -H “content-type:application/json” -X POST http://192.168.0.116:15672/api/bindings/%2f/e/webex/q/webq1 -d”{“”routing_key””:””webq1“”,””arguments””:{}}”

Comments: you need to bind a queue to an exchange if you wish to write to it. The routing key does the binding. The key can be any name but using the queue name makes it simple. A queue can also have binding to multiple exchanges.

Get a message: from queue webq1

curl -i -u pete:pete -H “content-type:application/json” -X POST http://192.168.0.116:15672/api/queues/%2f/webq1/get -d”{“”count””:5,””requeue””:true,””encoding””:””auto””,””truncate””:50000}”

Comments: A queue can have multiple messages. A requeue=false, will remove messages that are already read.

Post a message (44.4) : to exchange webex with routing key webq1

curl -i -u pete:pete -H “content-type:application/json” -X POST http://192.168.0.116:15672/api/exchanges/%2f/webex/publish -d”{“”properties””:{},””routing_key””:””webq1“”,””payload””:””44.4“”,””payload_encoding””:””string””}”

 

Javascript REST API Client

I had some Web browser issues when my test PC and my RabbitMQ server were in different domains. Web Browsers like Chrome and  Firefox will complain about a CORS policy problem.

Cross-origin resource sharing (CORS) is a mechanism that allows restricted resources on a web page to be requested from another domain outside the domain from which the first resource was served.

If you are crossing domains and you want to do some quick testing Microsoft Explorer will work.

Javascript GET a Queue Message

I used the AJAX XMLHttpRequest object to do my GET and POST requests.

<!DOCTYPE html>
<html>
<body>
<h2>RabbitMQ REST API - Get Queue Message</h2>

URL/API Call: <input type="text" id="theurl" value="http://192.168.0.116:15672/api/queues/%2f/"  size = "50" /> <br><br>
Queue Name:   <input type="text" id="qname" value="webex"/> <br><br>
Requeue: <input type="text" id="rq" value="true" /> (false = remove old values)<br><br>
Message Count: <input type="text" id="mcount" value="5"/> <br><br>
<button type="button" onclick="sendoutput()">Get Message(s)</button>
<br>

<hr>
<p id="demo"></p>

<script>
function sendoutput() {
	// RabbitMQ username and password
	var username = "pete";
	var password = "pete";
	var xhttp = new XMLHttpRequest();
	
    var fullurl = document.getElementById("theurl").value + document.getElementById("qname").value + "//get";
	
	xhttp.open("POST", fullurl, true,username,password);
	
	// change to your Auth username and password

	xhttp.setRequestHeader("Content-Type","application/jsonp");

	var rq = document.getElementById("rq").value;
	var mcount = document.getElementById("mcount").value;
	var params = JSON.stringify({"count":mcount,"requeue":rq,"encoding":"auto","truncate":"50000"});
	// Define a handler function when the response comes back
	xhttp.onreadystatechange = function() {
		// Get the response and put on it the screen
		if (this.readyState == 4 ) {	
			document.getElementById("demo").innerHTML = "Result=" +xhttp.responseText;
		}
	}

	xhttp.send(params);
}

</script>
</body>
</html>

r_get

Javascript POST an Exchange Message

Unfortunately you can not send a message directly to a queue, instead you need to send a message to an exchange with routing key. This routing key is linked to the queue that you wish to connect to.

<!DOCTYPE html>
<html>
<body>
<h2>RabbitMQ REST API - An Exchange Message</h2>

URL/API Call: <input type="text" id="theurl" value="http://192.168.0.116:15672/api/exchanges/%2f/"  size = "50" /> <br><br>
Exchange Name:   <input type="text" id="exname" value="webex"/> <br><br>
Routing Key:   <input type="text" id="rkey" value="webq1"/> <br><br>
Value to Send:   <input type="text" id="theval" value="11.1"/> <br><br>

<button type="button" onclick="sendoutput()">Send Output</button>
<br>

<hr>
<p id="demo"></p>

<script>
function sendoutput() {
	// RabbitMQ username and password
	var username = "pete"; 
	var password = "pete";  
	
	var xhttp = new XMLHttpRequest();
	
	var fullurl = document.getElementById("theurl").value + document.getElementById("exname").value + "//publish";
	
	xhttp.open("POST", fullurl, true,username,password);

	xhttp.setRequestHeader("Content-Type","application/jsonp");
	
	var rkey = document.getElementById("r").value
	var theval = document.getElementById("theval").value
	var params = JSON.stringify({"properties":{},"routing_key":rkey,"payload":theval,"payload_encoding":"string"});
	
	// Define a handler function when the response comes back
	xhttp.onreadystatechange = function() {
		// Get the response and put on it the screen
		if (this.readyState == 4 ) {	
			document.getElementById("demo").innerHTML = "Result=" +xhttp.responseText;
		}
	}
	xhttp.send(params);
}
</script>
</body>
</html>

r_post

Python – get message(s) from a queue

The requests library is extremely simple to use for REST interfacing. It is installed by:

pip install requests

# importing the requests library
import requests

# defining the api-endpoint
API_ENDPOINT = "http://192.168.0.116:15672/api/queues/%2f/webq1/get"

headers = {'content-type': 'application/json'}
# data to be sent to api
pdata = {'count':'5','requeue':'true','encoding':'auto','truncate':'50000'}

# sending post request and saving response as response object
r = requests.post(url = API_ENDPOINT ,auth=('pete', 'pete'), json = pdata, headers=headers)

# extracting response text
pastebin_url = r.text
print("Response :%s"%pastebin_url)

Python – Write a message to an exchange


# importing the requests library
import requests
import json

# defining the api-endpoint
API_ENDPOINT = "http://192.168.0.116:15672/api/exchanges/%2f/webex/publish"

# your source code here

headers = {'content-type': 'application/json'}
# data to be sent to api

pdata = {'properties':{},'routing_key':'webq1','payload':'90.4','payload_encoding':'string'}

# sending post request and saving response as response object
r = requests.post(url = API_ENDPOINT ,auth=('pete', 'pete'), json = pdata, headers=headers)

# extracting response text
pastebin_url = r.text
print("Response :%s"%pastebin_url)

Final Comments

The RabbitMQ REST interface is not designed for high throughput, but it is an excellent way to connect devices that have no native API support.

For my next round of testing I’d like to work on an Arduino and a AppInventor Android app.

Mosquitto MQTT Server

There are a number of excellent open source MQTT (Message Queue Telemetry Transport) brokers or servers that are available. Eclipse Mosquitto is a very popular light weight server that can be run on lower end hardware such as Raspberry Pi modules.

In this blog I’ll look at setting up a Mosquitto broker and I’ll setup some connections to it using testing tools and then use JavaScript with Web Sockets to look at Mosquitto diagnostics.

Getting Started

An MQTT project will have the following components:

  • MQTT Broker. The Mosquitto Server that manages and distributes messages
  • MQTT Publisher. A source for the data, such as a Arduino or Raspberry Pi
  • MQTT Subscriber. An application that wants the data

There is a good selection of MQTT libraries and test utilities for MQTT publishers and subscribers.

To install the Mosquitto broker and test utilities on a Raspberry Pi or Debian system enter:

sudo apt-get update
sudo apt-get install mosquitto mosquitto-clients

For Windows and other systems see the Mosquitto download documentation.

The features of the mosquitto broker are defined by the mosquitto configuration file (/etc/mosquitto/mosquitto.conf). For simple testing you can leave the configuration as is, however if you need to any added features like bridging of MQTT brokers, adding diagnostics or setting up MQTT for web access then you’ll need to edit mosquitto.conf.

For my project I wanted to add diagnostic (with a 10 second update), and I wanted MQTT to also run on Websockets. To do this I edited the config file by:

sudo nano /etc/mosquitto/mosquitto.conf

then I added the lines:

listener 8080
protocol websockets
sys_interval 10

The websockets can be set to any port number (9001, 8000,8080 etc.). After the file has been modified you’ll need to restart the Mosquitto service:

sudo service mosquitto restart

Test Clients

To test that your Mosquitto MQTT broker is running you can open 2 terminal sessions and run the mosquitto_sub and mosquitto_pub utilities.

In the first session we start mosquitto_sub with it subscribing to a topic called mytopic1:

~$ mosquitto_sub -t mytopic1
value1
1234

In the second session we use mosquitto_pub and we can publish some values to the topic mytopic1:

~$ mosquitto_pub -t mytopic1 -m "value1"
~$ mosquitto_pub -t mytopic1 -m 1234

If everything is working correctly then the published values should appear in the subscribing session.

JavaScript, Websockets and  Mosquitto Diagnostics

We can create a simple web page to test out the Javascript, Websockets and Mosquitto Diagnostics.

The Mosquitto Diagnostics have a topic of : $SYS/broker/… Some of these diagnostics include:

$SYS/broker/bytes/received
$SYS/broker/bytes/sent
$SYS/broker/clients/connected
$SYS/broker/clients/maximum
$SYS/broker/clients/total
$SYS/broker/heap/current size
$SYS/broker/heap/maximum size
$SYS/broker/load/connections/+
$SYS/broker/load/bytes/received/+
$SYS/broker/load/bytes/sent/+
$SYS/broker/load/messages/received/+
$SYS/broker/load/messages/sent/+
$SYS/broker/load/publish/dropped/+
$SYS/broker/load/publish/received/+
$SYS/broker/load/publish/sent/+
$SYS/broker/load/sockets/+
$SYS/broker/messages/received
$SYS/broker/messages/sent
$SYS/broker/publish/messages/received
$SYS/broker/publish/messages/sent
$SYS/broker/retained messages/count
$SYS/broker/store/messages/count
$SYS/broker/store/messages/bytes
$SYS/broker/subscriptions/count
$SYS/broker/version

where  + = 1min, 5min or 15min averages

For a full description of the Mosquitto diagnostics see: https://mosquitto.org/man/mosquitto-8.html .

Next we can create a web page where we can subscribe to some of these diagnostic topics. An MQTT javascript library can be remotely called from:

https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.js

This library is very similiar to Python Paho MQTT library, and it supports both subscribe and publish functions.

<html>
<head>
<title> MQTT Subscribe Message</title>
</head>
<script src="https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.js" type="text/javascript"></script>


<script>
// Define some MQTT variables


function sub_mqtt_msg() {
// Send an MQTT message
  var mqtt_server = document.getElementById("mqtt_server").value;
  var mqtt_port = Number(document.getElementById("mqtt_port").value);


  client = new Paho.MQTT.Client(mqtt_server, mqtt_port,"");
  client.onMessageArrived = onMessageArrived;
  client.onMessageArrived = onMessageArrived;
  client.connect({onSuccess:onConnect});
  document.getElementById("submsg").innerHTML = "Trying to connect...<br>";

}
function onConnect() {
  document.getElementById("submsg").innerHTML = "New connection made...<br>";
  var mqtt_destname = document.getElementById("mqtt_destname").value;  
  client.subscribe(mqtt_destname);
  document.getElementById("submsg").innerHTML = "Subscribing to topic: " + mqtt_destname + " ...<br>";
}
function onMessageArrived(message) {
  var result = message.destinationName + " : " + message.payloadString + "<br>";
  document.getElementById("submsg").innerHTML += result;
}

</script>
<body>
<h1>MQTT Subscribe Test Page</h1>

Server Name: <input type="text" id="mqtt_server" value="192.168.0.121"><br><br>
Websocket: <input type="text" id="mqtt_port" value="8080"><br><br>
DestinationName: <input type="text" id="mqtt_destname" value="$SYS/broker/#"><br><br>

<button onclick="sub_mqtt_msg()">Subscript to MQTT</button>
<hr>
<h2>Subscribed Messages:</h2>
<div id=submsg></div>
</body>

</html>

By subscribing to $SYS/broker/# we can get see all the different diagnostic values.

java_ws_diag2

Summary

Mosquitto is an excellent MQTT broker that can be run on both high end servers and low end hardware like Raspberry Pi’s.

It is important to note that with most MQTT libraries, (i.e. Python Paho, JavaScript and Node Red) the default setting is to not retain only values. This can be a problem if a new MQTT subscribers starts because it will not have any data until new values are published. To ensure that old values are available set the RETAIN=YES option on the MQTT publish call.

In this blog I wanted to examine, the diagnostic features of the Mosquitto MQTT broker, however there are many more features and functions of MQTT communications that need to be examined. There are some excellent MQTT guides on the Internet. I tried to document some of my notes on MQTT with Javascript and Charts, and MQTT with Arduino and Node Red.

 

 

 

 

 

 

DynamoDB with Python and JavaScript

Amazon Web Services (AWS) is a probably the best known Cloud services offering on the Internet.  At first glance AWS is a quite intimidating, learning about all the different features of AWS could be a full time job. For this blog I wanted to introduce one component of AWS called DynamoDB.

DynamoDB is a NoSQL database that could be connected to sensor hub devices such as Raspberry Pi’s for Internet of Things (IoT) solutions.

aws_services

Getting Started with DynamoDB

The first step to getting started with AWS is to create an account. You can open a free “testing” account but unfortunately you’ll need to show that you’re serious by including a credit card number.

After creating an account you can sign into the AWS Management Console (https://aws.amazon.com/console/). With a single user account it is possible to run different services in different locations. It is important to note that not all services are available in all locations, so select a location that offers all the services that you need.

AWS_Region

After a region is selected go to the Services menu item and search for DynamoDB, next select “create table”. A DynamoDB table has a NoSQL database structure, so this means that all that is required is a table name and a primary key. A primary key allows rows of data to have a way to be connected together.

For my IoT table example I created a primary key called signalname. (If you plan to have a very large data set and you need extremely high speed response you might want to consider defining multiple keys, such as location or sensor type).

create_table

Once the table is created, you can view and modify data.

the_table

The “Items” tab is used to add, view and delete data. Note, the viewing of updates is not automatic, so you need to press the “refresh” button periodically.

mytag1

Users and Security

The AWS user security is quite detailed. To get started select your account name on the menu bar and “My Security Credentials“.  The Groups item allows you to create security classifications for activities such as admin, viewing, etc.

New users can be defined and added into security groups.

user_security

When a new users is created an Access Key ID and a Secret Access Key is generated. The Secret Access Key is only shown when the user is created so ensure that you save it, and/or download the CSV file.

newuser

After an Access key ID and Secret Access Key is created for a user, you can start  programming interfaces to DynamoDB.

DynamoDB and Python

The Python AWS library is called boto3 and it is installed  in Windows by :

pip install boto3

and in Linux by:

sudo pip install boto3

The AWS security can be either added directly into the Python (or JavaScript) application or it can reside on the node or PC that you are working on. Both methods have pro’s and con’s. To add the credentials locally, you’ll need to install the AWS command line tool (awscli) and then enter your specific information:

$ sudo pip install awscli
$ aws configure
AWS Access Key ID [None]: AKIAIDxxxxx
AWS Secret Access Key [None]: kzzeko8Fxxxxxxxxxxxxxxxxx
Default region name [None]: Central
Default output format [None]:

A Python example to write and read a row into DynamoDB with the credentials in the program could be:

import boto3
import time

thesec = int(time.time() % 60)

mysession = boto3.session.Session(
    aws_access_key_id= 'AKIAID7xxxxxxxxx',
    aws_secret_access_key='kzzeko8xxxxxxxxxxxxxxxx')
db = mysession.resource('dynamodb', region_name='ca-central-1')

table = db.Table('my_iot_burl1')

response = table.put_item(
   Item={
        'signal': 'mytag1',
        'tagdesc': 'test tag1',
        'tagvalue': thesec
    })

response = table.get_item(Key={'signal': 'mytag1'})
print(thesec)
print(response)
print(response['Item']['tagvalue'])

If the credentials are locally used then the code to write data would be:

import boto3

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('users01')

table.put_item(
   Item={
        'signal': 'mytag1',
        'tagdesc': 'test tag1',
        'tagvalue': 16
    }
)

The response comes back as JSON data. For this example the result would be:

16
{‘Item’: {‘tagvalue’: Decimal(’16’), ‘tagdesc’: ‘test tag1’, ‘signal’: ‘mytag1’}, ‘ResponseMetadata’: {‘RequestId’: ’66M7R4CVCACA4KNA24LM90KG0NVV4KQNSO5AEMVJF66Q9ASUAAJG’, ‘HTTPStatusCode’: 200, ‘HTTPHeaders’: {‘server’: ‘Server’, ‘date’: ‘Fri, 15 Feb 2019 16:29:48 GMT’, ‘content-type’: ‘application/x-amz-json-1.0’, ‘content-length’: ’84’, ‘connection’: ‘keep-alive’, ‘x-amzn-requestid’: ’66M7R4CVCACA4KNA24LM90KG0NVV4KQNSO5AEMVJF66Q9ASUAAJG’, ‘x-amz-crc32’: ‘220267538’}, ‘RetryAttempts’: 0}}
16

For large amounts of data being written to DynamoDB it is possible to use a batch writing function:

import boto3
import time

db = boto3.resource('dynamodb', region_name='ca-central-1')

table = db.Table('my_iot_burl1')

thesec = int(time.time() % 60)

with table.batch_writer() as batch:
    for i in range(1,11):
        batch.put_item(
            Item={
                'signal': 'mytag' + str(i),
                'tagdesc': 'test tag'+ str(i),
                'tagvalue': thesec,
                'status': 'NORMAL'
            }
        )

The AWS console can be used to verify that the data was written in.

Writing to DynamoDB will clear the existing row with a completely new set of data. If you only want to update a field in a row then the update_item call is used. In the example below only the sensor tagvalue is updated and all other fields are left unchanged.

import boto3

mysession = boto3.session.Session(
    aws_access_key_id= 'AKIAIDxxxxxxxxx',
    aws_secret_access_key='kzzekxxxxxxxxxxxxxxx'
)

db = mysession.resource('dynamodb', region_name='ca-central-1')

table = db.Table('my_iot_burl1')

# set the value for mytag1 to bet 29
table.update_item(
    Key={ 'signal': 'mytag1' },
    UpdateExpression='SET tagvalue = :newval1',
    ExpressionAttributeValues={
        ':newval1': 29
    }
)

Python “Scan the Data” Example

The boto3 library supports the ability to scan the data based on some filtering criteria. The filter syntax is: Attr(‘the field’).eq(‘thevalue’).

The following comparison operators are available:

eq | ne | le | lt | ge | gt | not_null | null | contains | not_contains | begins_with | in | between

Multiple statements can be created with | (OR) , & (AND) operators. Below is an example that scans the database for rows with an alarm status or with a tag value greater than 100.

scan_view

import boto3
from boto3.dynamodb.conditions import Key, Attr

db = boto3.resource('dynamodb', region_name='ca-central-1')

table = db.Table('my_iot_burl1')

response = table.scan(
    FilterExpression=Attr('status').eq('ALARM') | Attr('tagvalue').gt(100)
)

#print (response)
for i in response['Items']:
    print(i)
    print(i['signal'], "=", i['tagvalue'])

The results were:

{‘tagvalue’: Decimal(‘150’), ‘signal’: ‘mytag2’, ‘tagdesc’: ‘test tag2’, ‘status’: ‘NORMAL’}
mytag2 = 150
{‘tagvalue’: Decimal(‘999’), ‘signal’: ‘mytag1’, ‘tagdesc’: ‘test tag1’, ‘status’: ‘ALARM‘}
mytag1 = 999

JavaScript

For standard HTML/JavaScript projects the credentials are included in the source file. This obviously has some serious security issues, so it’s recommended that you define a low level user that can only do a small subset of functions.

Below is a simple example that can read our DynamoDB table and it can write a new row with an updated value.

<html>
<head>
<script src="https://sdk.amazonaws.com/js/aws-sdk-2.7.16.min.js"></script>

<script>
AWS.config.update({
  region: "ca-central-1",
  accessKeyId: "AKIAID7xxxxxxxxxxxx",
  secretAccessKey: "kzzeko8xxxxxxxxxxxxx"
});

var docClient = new AWS.DynamoDB.DocumentClient();

function createItem() {
  var d = new Date();
  var n = d.getSeconds();
    var params = {
        TableName :"my_iot_burl1",
        Item:{
      'signal': 'mytag4',
      'desc': 'test tag4',
      'info': {
        'value':n,
        'quality': "questionable"
        }
    }
    };
    docClient.put(params, function(err, data) {
        if (err) {
            document.getElementById('textarea').innerHTML = "Unable to add item: " + "\n" + JSON.stringify(err, undefined, 2);
        } else {
            document.getElementById('textarea').innerHTML = "PutItem succeeded: " + "\n" + JSON.stringify(params,undefined, 2);
        }
    });
}

function readItem() {
    var table = "my_iot_burl1";
    var signal = 'mytag4';
    var params = {
        TableName: table,
        Key:{'signal' : 'mytag4' }
    };
    docClient.get(params, function(err, data) {
        if (err) {
            document.getElementById('textarea2').innerHTML = "Unable to read item: " + "\n" + JSON.stringify(err, undefined, 2);
        } else {
            document.getElementById('textarea2').innerHTML = "GetItem succeeded: " + "\n" + JSON.stringify(data, undefined, 2);
        }
    });
}

</script>
</head>

<body>
<input id="createItem" type="button" value="Create Item" onclick="createItem();" />
<br><br>
<textarea readonly id= "textarea" style="width:400px; height:300px"></textarea>
<hr>
<input id="readItem" type="button" value="Read Item" onclick="readItem();" />
<br><br>
<textarea readonly id= "textarea2" style="width:400px; height:300px"></textarea>
</body>
</html>

aws_html

Summary

On first impressions I was pretty overwhelmed using Amazon Web Services. However after some time playing with it I found that it wasn’t unlike many other Web services that are available.

DynamoDB can be used to communicate with sensor values from a Raspberry Pi, but unfortunately there are no mainstream libraries for Arduino projects. AWS does offer an MQTT solution that can be used to update a DynamoDB table.

For small IoT projects I feel that AWS is overkill, but for larger projects I would definitely consider using it.

Javascript talking to littleBits Cloud API

LittleBits is a set of components that allow kids to build their own electrical circuits. The Cloudbit will send and receive values from the Internet.

For this project we wanted to build some Web pages with Javascript that could interact with the littleBits Cloud API.

littleBits Hardware Setup

cloudbit_setup

The Cloudbit enables an input to be sent to the Internet, and it will receive a value (output) from the Internet. For a test setup we used a dimmer bit to adjust the input value with a number bit to show the value. On the output side we used a bargraph bit. Other combinations of bits are possible. The key is to be able to test and see your input and output values.

Cloudbits Setup

For the Cloudbit setup you will need to create a sign-in on the littleBits web site. For details on how to setup your Cloudbit use the following link.

The Cloud API is a RESTful interface, that uses http requests with some header definitions. Before you get started you’ll need your specific littleBits device ID and authorization token. In your specific Cloudbit definition go to the Settings icon, and get your Device ID and AccessToken.

littlebit_id

Cloudbit Rest API – First Example Testing with curl

The Cloudbit API reference can be found at : http://developers.littlebitscloud.cc/#devices. 

The Cloudbit API is based on the REST API standard, and this means that an HTTP request is made with some extra parameters that are passed in the header. The most important header item is the littleBits authorization token.

Curl is command line software tool that allows you to issue an HTTP request with header and method parameters and it returns the results in a text format. Curl is available for Windows, OSX and Linux. As an example to query the CloudBit API to get all my devices the following command is issued (note you’ll need to enter your own authorization code) :

curl "https://api-http.littlebitscloud.cc/v2/devices" \
-H "Authorization: Bearer 4f3830b44e1d4b2789b50b0xxxxxx"

[
{"label":"twinbit",
"id":"00e04c0379bb",
"subscriptions":[],
"subscribers":[],
"user_id":118217,
"is_connected":true,
"input_interval_ms":200}
]

Javascript Device Monitor

The curl example can be done in Javascript using the Javascript built in XMLHttpRequest object.

A monitor page can be useful if you have a number of Cloudbits and you’re interested is checking their status. Below is a sample web page and the associated HTML/javascript code. It’s important to note that text that is returned needs to have the first few characters removed in order to have a clean JSON string.

js_status

<!DOCTYPE html>
<html>
<body>
<h1 id='title'>Cloudbit Status Monitor</h1>
<h2>
Label : <font color='red' id='LB_label'></font><br>
Is Connected : <font color='red' id='is_connected'></font><br>
</h2>
 
<script>
var xhttp = new XMLHttpRequest();
// change the authtoken to match your settings
var authtoken = "4f3830b44e1dxxxxxxxxxxxxxxxx";
var theURL =  "https://api-http.littlebitscloud.cc/devices/";
xhttp.open("GET", theURL, true);
xhttp.setRequestHeader("Accept","application/vnd.littlebits.v2+json");
xhttp.setRequestHeader("Authorization", "Bearer " + authtoken);
xhttp.onreadystatechange = function() {
  if (xhttp.readyState == 4 ) { // when the response is complete get the data
    // remove leading "[" and trailing ']'
    var theresponse = xhttp.responseText.substring(1, xhttp.responseText.length -1);
    var lb_data = JSON.parse( theresponse );  
    
    document.getElementById('LB_label').innerText =  lb_data.label;
    document.getElementById('is_connected').innerHTML = lb_data.is_connected;
  }
}
xhttp.send();
</script>

</body>


</html>

Read Value Example

Reading a CloudBit value is a little tricky because the data is returned as a stream that updates every second. I saw a lot of people asking for a single point read, so hopefully this will be available soon.

The read value HTTP request is a little more complex than the simple monitor device example because you need to include the device that you are querying and a head parameter of :  Accept: application/json.

A curl example would be:

curl -XGET "https://api-http.littlebitscloud.cc/v2/devices/00e04c0379bb/input" -H "Authorization: Bearer 4f3830b44e1d4b2789b50b09cb493f06750b968cff5d45331c75006025fa0dc9" -H Accept: application/vnd.littlebits.v2+json"

data:{"type":"input","timestamp":1543536039623,"percent":6,"absolute":66,"name":"amplitude","payload":{"percent":6,"absolute":66},"from":{"device":{"id":"00e04c0379bb","mac":"00e04c0379bb"}}}

data:{"type":"input","timestamp":1543536039650,"percent":6,"absolute":65,"name":"amplitude","payload":{"percent":6,"absolute":65},"from":{"device":{"id":"00e04c0379bb","mac":"00e04c0379bb"}}}

...

For an HTML/Javascript single point read the key to check the xhttp.readyState == 3 , this will catch the first streamed response.  Below is a single point example with code.

js_in

<!DOCTYPE html>
<html>
<head>
<title>littleBits Get Input</title>
<script>
var theinput = 0;

function get_input() {
// update with your deviceid and authtoken
  var deviceid = "00e04c0379bb";
  var authtoken = "4f3830b44e1d4b27xxxx";
  var theurl = "https://api-http.littlebitscloud.cc/devices/";
  var xhttp = new XMLHttpRequest();
  xhttp.onreadystatechange = function() {

    if (xhttp.readyState == 3 ) {
	  	var datapackage = xhttp.responseText.split("\n\ndata:");
		var lb_data = JSON.parse( datapackage[1] );		 
		document.getElementById("thevalue").innerText =  lb_data.percent;
		xhttp.open("GET","",true);
		xhttp.send();
    }
  }
  xhttp.open("GET", theurl + deviceid + "/input", true);
  xhttp.setRequestHeader("Accept","application/json");
  xhttp.setRequestHeader("Authorization", "Bearer " + authtoken);
  xhttp.send();
}
</script>
</head>
<body>

<h1 id='title'>littleBit Get Input</h1>
The value : <font id="thevalue"> XXXX </font>
<button type="button" onclick="get_input()">Request data</button>
<br>

</body>
</html>

Output Example

A CloudBit output can be either sustained or it can be timed out. The curl command is:

curl "https://api-http.littlebitscloud.cc/v2/devices/yourdevice_id/output" \
  -X POST \
  -H "Authorization: your_auth_code" \
  -H "Content-type: application/json" \
  -d '{ "percent": 100, "duration_ms": 3000 }'

For an HTML/Javascipt web the http response is a POST, and the value’s percent and duration are put into a JSON string:

var params = JSON.stringify({duration_ms: duration,percent: thevalue});

An example page and code is shown below:

js_out

<!DOCTYPE html>
<html>
<body>
<h2>CloudBit Output Test</h2>
<button type="button" onclick="sendoutput()">Send Output</button>
<br>
<pre>
Output Time (ms): <input type="text" id="duration" value="-1"/> (constant = -1)</br>
Output Value	: <input type="text" id="thevalue" value="80"/> (percent 0-100)</br> 
</pre>
<p id="demo"></p>


function sendoutput() {
	var xhttp = new XMLHttpRequest();
        // change to your device id
	xhttp.open("POST", "https://api-http.littlebitscloud.cc/devices/00e04c0379bb/output?", true);
	xhttp.setRequestHeader("Accept","application/vnd.littlebits.v2+json");
	// change to your Auth Token
	xhttp.setRequestHeader("Authorization", "Bearer 4f3830b44e1d4b27xxxx");
	xhttp.setRequestHeader("Content-Type","application/json");

	var duration = document.getElementById("duration").value;
	var thevalue = document.getElementById("thevalue").value;

	var params = JSON.stringify({duration_ms: duration,percent: thevalue});

	xhttp.onreadystatechange = function() {
		document.getElementById("demo").innerHTML = "Result=" +xhttp.responseText;
	}

	xhttp.send(params);
}


</body>
</html>

Gauges

Once you’ve got the basics down it’s possible to start making some more advanced applications. There are lot of Javascript charting libraries. For this example I used the Google  Charts library.

js_gauge

<html>
  <head>
   https://www.gstatic.com/charts/loader.js
   
	  var littleBitinput = 0;
      google.charts.load('current', {'packages':['gauge']});
      google.charts.setOnLoadCallback(drawChart);
      function drawChart() {

        var data = google.visualization.arrayToDataTable([
          ['Label', 'Value'],
          ['littleBits', 100],
        ]);

        var options = { 
          width: 600, height: 600,
          redFrom: 90, redTo: 100,
          yellowFrom:75, yellowTo: 90,
		  greenFrom: 60, greenTo: 75,
          minorTicks: 5
        };

        var chart = new google.visualization.Gauge(document.getElementById('chart_div'));

        chart.draw(data, options);

        setInterval(function() {
		  get_input();
          data.setValue(0, 1, littleBitinput);
          chart.draw(data, options);
        }, 500);
 
      }
	  function get_input() {
	  // change to your deviceid and authtoken
		  var deviceid = "00e04c0379bb";
		  var authtoken = "4f3830b44e1d4b27xxx";
		  var theurl = "https://api-http.littlebitscloud.cc/devices/";		  
		  var xhttp = new XMLHttpRequest();
		  
		  xhttp.onreadystatechange = function() {
			if (xhttp.readyState == 3 ) {
				var datapackage = xhttp.responseText.split("\n\ndata:");
				var lb_data = JSON.parse( datapackage[1] );		 
				littleBitinput =  lb_data.percent;
				xhttp.open("GET","",true);
				xhttp.send();
			}
		  }
		  xhttp.open("GET", theurl + deviceid + "/input", true);
		  xhttp.setRequestHeader("Accept","application/vnd.littlebits.v2+json");
		  xhttp.setRequestHeader("Authorization", "Bearer " + authtoken);
		  xhttp.send();

}
    
  </head>
  <body>

 

Final Comment

There are some great projects that you could do with littleBits and the CloudAPI.

One still project I did was to use smoothie charts http://smoothiecharts.org/ for the real time charts.

lb_chart2

 

 

 

 

MQTT and Javascript

MQTT (Message Queuing Telemetry Transport) is a  publish-subscribe-based messaging protocol that is used on many Internet of Things (IoT) projects. It works on top of the TCP/IP protocol and it is designed for connections with remote locations where a “small code footprint” is required or the network bandwidth is limited. The publish-subscribe messaging pattern requires a message broker.

MQTT_js_overview

There is support for MQTT on a large variety of programming languages and platforms. An Arduino or Raspberry Pi module can sent (or publish) I/O to a MQTT broker, and they can also receive (or subscribe) to data.

There are a number of MQTT brokers that can be used. One of the most popular ones is the Mosquitto MQTT broker, and it can be loaded on Windows, OSX and Linux systems. For this blog we will be using the Mosquitto test MQTT server. This Internet based server should not be used for real systems, but it is excellent for small short terms tests.

MQTT Web Sockets

The MQTT server has configurable MQTT and Web Socket ports. For a typical Raspberry Pi or Arduino connection, the default MQTT port 1883 would be used. In many Internet applications only certain ports are open, so for this reason a different MQTT Web Socket is used. This is configurable but ports like 80 or 8080 are typically used.MQTT_web_layout

Javascript Application

There are a number of MQTT javascript libraries that are available. One of the more popular ones is the Paho library that is available at:

https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.js

As a first example we will look at creating 2 pages. The first page will  publish a value, and the second page will subscribe to the data.

MQTT_js_pubsub

The publish code is:

<html>
<head>
<title> MQTT Publish Message</title>
</head>
<script src="https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.js" type="text/javascript"></script>

<script>
// Define some MQTT variables
var mqtt_server = "";
var mqtt_port = "";
var mqtt_destname = "";

function send_mqtt_msg() {
// Send an MQTT message
  mqtt_server = document.getElementById("mqtt_server").value;
  mqtt_port = Number(document.getElementById("mqtt_port").value);
 

  client = new Paho.MQTT.Client(mqtt_server, mqtt_port,"");
  client.connect({onSuccess:onConnect});
  document.getElementById("pubmsg").innerHTML = "Trying to connect...
";
}
function onConnect() {
  document.getElementById("pubmsg").innerHTML = "New connection made...
";
  var mqtt_destname = document.getElementById("mqtt_destname").value;
  var mqtt_msg = document.getElementById("mqtt_msg").value;   
  message = new Paho.MQTT.Message(mqtt_msg);
  message.destinationName = mqtt_destname;
  client.send(message);
  document.getElementById("pubmsg").innerHTML = "topic:" + mqtt_destname + " = " + mqtt_msg + " ...sent";
}  
// called when a message arrives
</script>
<body>
<h1>MQTT Publish Test Page</h1>

Server Name: <input type="text" id="mqtt_server" value="test.mosquitto.org"><br><br>
Websocket: <input type="text" id="mqtt_port" value="8080"><br><br>
DestinationName: <input type="text" id="mqtt_destname" value="my_IoT_value1"><br><br>
Message: <input type="text" id="mqtt_msg" value="test message"><br><br>

<button onclick="send_mqtt_msg()">Publish MQTT Message</button>
</body>
<hr>
<div id=pubmsg></div>
</html>

The subscribe code is:

<html>
<head>
<title> MQTT Subscribe Message</title>
</head>
<script src="https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.js" type="text/javascript"></script>
<script>

function sub_mqtt_msg() {
// Send an MQTT message
  var mqtt_server = document.getElementById("mqtt_server").value;
  var mqtt_port = Number(document.getElementById("mqtt_port").value);

  client = new Paho.MQTT.Client(mqtt_server, mqtt_port,"");
  client.onMessageArrived = onMessageArrived;
  client.onMessageArrived = onMessageArrived;
  client.connect({onSuccess:onConnect});
  document.getElementById("submsg").innerHTML = "Trying to connect...
";

}
function onConnect() {
  document.getElementById("submsg").innerHTML = "New connection made...
";
  var mqtt_destname = document.getElementById("mqtt_destname").value;  
  client.subscribe(mqtt_destname);
  document.getElementById("submsg").innerHTML = "Subscribing to topic: " + mqtt_destname + " ...
";
}
function onMessageArrived(message) {
  var result = message.destinationName + " : " + message.payloadString + "
";
  document.getElementById("submsg").innerHTML = result;
}

</script>
<body>
<h1>MQTT Subscribe Test Page</h1>

Server Name: <input type="text" id="mqtt_server" value="test.mosquitto.org"><br><br>
Websocket: <input type="text" id="mqtt_port" value="8080"><br><br>
DestinationName: <input type="text" id="mqtt_destname" value="my_IoT_value1"><br><br>

<button onclick="sub_mqtt_msg()">Subscript to MQTT</button>
<hr>
<h2>Subscribed Messages:</h2>
<div id=submsg></div>
</body>

</html>

Once you’ve got the basics down it’s possible to make some more advanced web interface pages.

Chart and Gauges

For your IoT projects there are a lot of Dash board options that are available. One of my favorites is Node-Red because it is totally free and standalone.

If you would like to create your own web interface there are a number of good javascript charting and gauge libraries available. For my examples I used Google charts with the Gauge chart . To simulate the inputs I used three of our MQTT Publish pages.

MQTT_js_sim

The code for the gauges page is :

<html>
<head>
<title>IoT - MQTT to JS</title>
</head>
<script src="https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.js" type="text/javascript"></script>
<script src="https://www.gstatic.com/charts/loader.js" type="text/javascript"></script>

<script>
// MQTT variables
var MQTTnames = ["my_IoT_value1","my_IoT_value2","my_IoT_value3"];
var MQTTvalues = [0,0,0];


// Define the Google gauge chart
      google.charts.load('current', {'packages':['gauge']});
      google.charts.setOnLoadCallback(drawChart);

      function drawChart() {

        var data = google.visualization.arrayToDataTable([
          ['Label', 'Value'], 
      [MQTTnames[0], MQTTvalues[0]],
      [MQTTnames[1], MQTTvalues[1]],
      [MQTTnames[2], MQTTvalues[2]],      
        ]);

        var options = {
          width: 800, height: 1000,
          redFrom: 90, redTo: 100,
          yellowFrom:75, yellowTo: 90,
          minorTicks: 5
        };

        var chart = new google.visualization.Gauge(document.getElementById('chart_div'));

        chart.draw(data, options);

        setInterval(function() {
      for (var i=0; i < MQTTnames.length; i++) {
      data.setValue(i, 1, MQTTvalues[i]);
      }
          chart.draw(data, options);

      }, 1000);
}

// Create a client instance
client = new Paho.MQTT.Client("test.mosquitto.org", 8080,"");
client.onMessageArrived = onMessageArrived;

// connect the client
client.connect({onSuccess:onConnect});

// called when the client connects
function onMessageArrived(message) {
  
  for (var i=0; i < MQTTnames.length; i++) {
  if (message.destinationName == MQTTnames[i]) {
    MQTTvalues[i] = Number(message.payloadString);
  }
  }
}
function onConnect() {
  // Once a connection has been made, make a subscription and send a message.
  for (var i=0; i < MQTTnames.length; i++) {
  client.subscribe(MQTTnames[i]);
  }
}
// called when a message arrives
</script>
<body>
<h1  style = 'font-size: xx-large'>IoT - MQTT to JavaScript</h1>
<div id="chart_div" style="width: 100%; height: 100%;"></div>
</body>
</html>

There are many other charting options that could be used. Below is an example using the Google Chart library with bars instead of gauges.

MQTT_js_bars.png

Final Comment

Using Javascript to interact with your IoT projects open up a lot of potential for adding functionality. I found that the Python version of the Paho MQTT library to have better documentation and perhaps some more functions, but at the end of the day I was able to do all the I wanted.