RabbitMQ and Excel

RabbitMQ is one of the most popular messaging queue systems out there. If you’re a programmer there are client packages available for almost of the programming languages. However if you’re not a programmer your choices are limited in trying to interact with RabbitMQ.

In this blog I wanted to document some of my work in using Excel to read and write messages from RabbitMQ.

Getting Started

There are some great RabbitMQ guides. To get Excel talking to RabbitMQ I used the REST API that will be need to be installed.

For my Excel work I used Visual Basic for Applications (VBA) that is launched by Alt-F11. Next I created a Module (this is a container for VBA code), and I created two functions:

  • RMQ_getqueue – to get the first value in a queue
  • RMQ_exchangeWrite – to write a value to an exchange  (note: if an exchange has a binding to a queue. Writing to an exchange will be also be writing to a queue).

VBA_insert_module

Read Message Queues

For my example I focused on getting the first item in a queue. Message in a queue can be cleared as they are read if the option requeue = False.

 Function RMQ_getqueue(RabbitMQ_Server As String, QueueName As String, Optional rmq_user As String, Optional rmq_pswd As String, Optional requeue As Boolean)  
 ' Create a REST API get request to a Rabbitmq server and get the first item in a queue  
   Dim theJSON As Object, http As Object, scriptControl As Object  
   Dim theurl As String, theparams As String, rqmsg As String  
   Set http = CreateObject("MSXML2.XMLHTTP")  
 ' Use a Jscript eval command to evaluate the JSON, (so you don't have to add a JSON component  
   Set scriptControl = CreateObject("MSScriptControl.ScriptControl")  
   scriptControl.Language = "JScript"  
     
   theurl = "http://" & RabbitMQ_Server & "/api/queues/%2f/" & QueueName + "/get"  
    
 ' define a parameter list this must be JSON and the synax is pickie  
   If IsMissing(requeue) Then requeue = True  
   theparams = "{""count"":1,""requeue"":""" & LCase(requeue) & """,""encoding"":""auto"",""truncate"":50000}"  
   'Debug.Print theparams  
     
   http.Open "POST", theurl, False, rmq_user, rmq_pswd  
   http.setRequestHeader "Content-type", "application/xml"  
   http.send (theparams)  
 ' create a simple JSON string  
   thestr = Mid(http.responseText, 2, Len(http.responseText) - 2)  
   Set theJSON = scriptControl.Eval("(" & thestr & ")")  
   'Debug.Print http.responseText  
   
   RMQ_getqueue = theJSON.payload  
 End Function  

The RMQ_getqueue function creates a URL string that is passed to the http request object. The RabbitMQ response text is JSON formatted. I didn’t want to load external JSON addins so I used the ScriptControl with a Javascript eval function to parse the JSON. The payload item of the response is the first message in the queue.

To use this new function in Excel select the category “User Defined”.

Excel_insert_function

Then fill in the parameters as per your requirement.

Excel_rmq_get

Write a Message to an Exchange

As mentioned earlier you can’t write directly to a queue with the REST API instead you write to an exchange. The exchange has a routing key that “binds” a messages to a queue.

Like the get queue function the RMQ_exchangeWrite function formats a URL with all the required parameters.

 Function RMQ_exchangeWrite(RabbitMQ_Server As String, ExchangeName As String, RouteKey As String, theMessage As String, Optional rmq_user As String, Optional rmq_pswd As String)  
 ' Create a REST API post a new message to a Rabbitmq Exchange with a Routing Key  
   Dim http As Object  
   Dim theurl As String, theparams As String  
   Set http = CreateObject("MSXML2.XMLHTTP")  
     
   theurl = "http://" & RabbitMQ_Server & "/api/exchanges/%2f/" & ExchangeName + "/publish"  
    
 ' define a parameter list this must be JSON and the synax is pickie  
   theparams = "{""properties"":{},""routing_key"":""" & RouteKey & """,""payload"":""" & theMessage & """,""payload_encoding"":""string""}"  
     
   Debug.Print theparams  
     
   http.Open "POST", theurl, False, rmq_user, rmq_pswd  
   http.setRequestHeader "Content-type", "application/xml"  
   http.send (theparams)  
   
   'Debug.Print http.responseText  
   
   RMQ_exchangeWrite = http.statusText & " " & http.responseText  
 End Function  

Like the earlier function, fill in all the parameter as per your system.

Excel_rmq_write

Final Comments

This was an introductory project, and there are still a lot of features and error checking that should be included.

If I had more time I would put some dialogs in to dynamically show queues, exchanges etc.

RabbitMQ REST API

RabbitMQ is an open source middleware message solution that natively uses AMQP communications but it has a good selection of plug-ins to support features like: MQTT, MQTT Web Sockets, HTTP REST API and server-to-server communications.

rabbitmq_overview

RabbitMQ has a very good selection of interfacing clients for most of the major programming languages. For applications where there is no custom programming library the REST API can be used.

Applications where the REST API could be used might include:

  • Arduino IoT projects
  • Web Clients (not using NodeJS)
    • For simple reading and writing of messages
    • For monitoring of RabbitMQ loading
  • AppInventor Android apps

Unfortunately I didn’t find the RabbitMQ REST API to be 100% straightforward, so in this blog I could like to document some of my notes and testing.

What is REST API

Representational State Transfer (REST) is a software style for creating Web services. A client issues an HTTP request to a server with a GET/POST/PUT or DELETE method and the server responds with a JSON (Javascript Object Notation) formatted message.

RESTAPI_overview

The most common tool to test REST API is curl. Curl is supported on all major OS’s. The curl syntax will vary slightly between Windows and Linux, especially in how the systems handle strings.

RabbitMQ Setup

RabbitMQ can be installed on Window, Linux, MacOS systems and there are also some cloud based offerings. For small systems lower end hardware like a Raspberry Pi can be used.  For complete RabbitMQ installation instructions see: https://www.rabbitmq.com/download.html . To install and run RabbitMQ on a Ubuntu system enter:

sudo apt-get update
sudo apt-get install rabbitmq-server
sudo service rabbitmq-server start

The next step is to add some plug-ins. For the Web Administration and REST API plug-ins:

sudo rabbitmq-plugins enable rabbitmq-management

The rabbitmqctl command line tool allows you to configure and review the RabbitMQ server. To add a user pete,  with password pete, that has config, write and read rights for management and administrator access, enter:

sudo rabbitmqctl add_user pete pete
sudo rabbitmqctl set_permissions -p / pete ".*" ".*" ".*"
sudo rabbitmqctl set_user_tags pete management administrator

After you’ve defined an administrative user the RabbitMQ web management plug-in can be accessed by: http://ip_address:15672 .

Rabbit_webadmin

Once the Web Management component is installed it is much easier to test and play with REST API calls.

REST API Calls Using CURL

For a complete set of RabbitMQ REST API calls see: https://pulse.mozilla.org/api/ .

Typically you would use the RabbitMQ tools to create and exchanges and queues but it is possible to do this via the REST API interface.

Below are some examples using user: pete and a password of pete . Note: %2f is used for default (blank name) virtual host.

Create an exchange: webex

curl -i -u pete:pete -H “content-type:application/json” -X PUT http://192.168.0.116:15672/api/exchanges/%2f/webex -d”{“”type””:””direct””,””auto_delete””:false,””durable””:true,””internal””:false,””arguments””:{}}”

Create a Queue: webq1

curl -i -u pete:pete -H “content-type:application/json” -X PUT http://192.168.0.116:15672/api/queues/%2f/webq1 -d”{“”auto_delete””:false,””durable””:true,””arguments””:{}}”

Create Binding: Bind queue webq1 to exchange webex with a routing key of webq1

curl -i -u pete:pete -H “content-type:application/json” -X POST http://192.168.0.116:15672/api/bindings/%2f/e/webex/q/webq1 -d”{“”routing_key””:””webq1“”,””arguments””:{}}”

Comments: you need to bind a queue to an exchange if you wish to write to it. The routing key does the binding. The key can be any name but using the queue name makes it simple. A queue can also have binding to multiple exchanges.

Get a message: from queue webq1

curl -i -u pete:pete -H “content-type:application/json” -X POST http://192.168.0.116:15672/api/queues/%2f/webq1/get -d”{“”count””:5,””requeue””:true,””encoding””:””auto””,””truncate””:50000}”

Comments: A queue can have multiple messages. A requeue=false, will remove messages that are already read.

Post a message (44.4) : to exchange webex with routing key webq1

curl -i -u pete:pete -H “content-type:application/json” -X POST http://192.168.0.116:15672/api/exchanges/%2f/webex/publish -d”{“”properties””:{},””routing_key””:””webq1“”,””payload””:””44.4“”,””payload_encoding””:””string””}”

 

Javascript REST API Client

I had some Web browser issues when my test PC and my RabbitMQ server were in different domains. Web Browsers like Chrome and  Firefox will complain about a CORS policy problem.

Cross-origin resource sharing (CORS) is a mechanism that allows restricted resources on a web page to be requested from another domain outside the domain from which the first resource was served.

If you are crossing domains and you want to do some quick testing Microsoft Explorer will work.

Javascript GET a Queue Message

I used the AJAX XMLHttpRequest object to do my GET and POST requests.

<!DOCTYPE html>
<html>
<body>
<h2>RabbitMQ REST API - Get Queue Message</h2>

URL/API Call: <input type="text" id="theurl" value="http://192.168.0.116:15672/api/queues/%2f/"  size = "50" /> <br><br>
Queue Name:   <input type="text" id="qname" value="webex"/> <br><br>
Requeue: <input type="text" id="rq" value="true" /> (false = remove old values)<br><br>
Message Count: <input type="text" id="mcount" value="5"/> <br><br>
<button type="button" onclick="sendoutput()">Get Message(s)</button>
<br>

<hr>
<p id="demo"></p>

<script>
function sendoutput() {
	// RabbitMQ username and password
	var username = "pete";
	var password = "pete";
	var xhttp = new XMLHttpRequest();
	
    var fullurl = document.getElementById("theurl").value + document.getElementById("qname").value + "//get";
	
	xhttp.open("POST", fullurl, true,username,password);
	
	// change to your Auth username and password

	xhttp.setRequestHeader("Content-Type","application/jsonp");

	var rq = document.getElementById("rq").value;
	var mcount = document.getElementById("mcount").value;
	var params = JSON.stringify({"count":mcount,"requeue":rq,"encoding":"auto","truncate":"50000"});
	// Define a handler function when the response comes back
	xhttp.onreadystatechange = function() {
		// Get the response and put on it the screen
		if (this.readyState == 4 ) {	
			document.getElementById("demo").innerHTML = "Result=" +xhttp.responseText;
		}
	}

	xhttp.send(params);
}

</script>
</body>
</html>

r_get

Javascript POST an Exchange Message

Unfortunately you can not send a message directly to a queue, instead you need to send a message to an exchange with routing key. This routing key is linked to the queue that you wish to connect to.

<!DOCTYPE html>
<html>
<body>
<h2>RabbitMQ REST API - An Exchange Message</h2>

URL/API Call: <input type="text" id="theurl" value="http://192.168.0.116:15672/api/exchanges/%2f/"  size = "50" /> <br><br>
Exchange Name:   <input type="text" id="exname" value="webex"/> <br><br>
Routing Key:   <input type="text" id="rkey" value="webq1"/> <br><br>
Value to Send:   <input type="text" id="theval" value="11.1"/> <br><br>

<button type="button" onclick="sendoutput()">Send Output</button>
<br>

<hr>
<p id="demo"></p>

<script>
function sendoutput() {
	// RabbitMQ username and password
	var username = "pete"; 
	var password = "pete";  
	
	var xhttp = new XMLHttpRequest();
	
	var fullurl = document.getElementById("theurl").value + document.getElementById("exname").value + "//publish";
	
	xhttp.open("POST", fullurl, true,username,password);

	xhttp.setRequestHeader("Content-Type","application/jsonp");
	
	var rkey = document.getElementById("r").value
	var theval = document.getElementById("theval").value
	var params = JSON.stringify({"properties":{},"routing_key":rkey,"payload":theval,"payload_encoding":"string"});
	
	// Define a handler function when the response comes back
	xhttp.onreadystatechange = function() {
		// Get the response and put on it the screen
		if (this.readyState == 4 ) {	
			document.getElementById("demo").innerHTML = "Result=" +xhttp.responseText;
		}
	}
	xhttp.send(params);
}
</script>
</body>
</html>

r_post

Python – get message(s) from a queue

The requests library is extremely simple to use for REST interfacing. It is installed by:

pip install requests

# importing the requests library
import requests

# defining the api-endpoint
API_ENDPOINT = "http://192.168.0.116:15672/api/queues/%2f/webq1/get"

headers = {'content-type': 'application/json'}
# data to be sent to api
pdata = {'count':'5','requeue':'true','encoding':'auto','truncate':'50000'}

# sending post request and saving response as response object
r = requests.post(url = API_ENDPOINT ,auth=('pete', 'pete'), json = pdata, headers=headers)

# extracting response text
pastebin_url = r.text
print("Response :%s"%pastebin_url)

Python – Write a message to an exchange


# importing the requests library
import requests
import json

# defining the api-endpoint
API_ENDPOINT = "http://192.168.0.116:15672/api/exchanges/%2f/webex/publish"

# your source code here

headers = {'content-type': 'application/json'}
# data to be sent to api

pdata = {'properties':{},'routing_key':'webq1','payload':'90.4','payload_encoding':'string'}

# sending post request and saving response as response object
r = requests.post(url = API_ENDPOINT ,auth=('pete', 'pete'), json = pdata, headers=headers)

# extracting response text
pastebin_url = r.text
print("Response :%s"%pastebin_url)

Final Comments

The RabbitMQ REST interface is not designed for high throughput, but it is an excellent way to connect devices that have no native API support.

For my next round of testing I’d like to work on an Arduino and a AppInventor Android app.

Apache Kafka with Node-Red

Apache Kafka is a distributed streaming and messaging system. There are a number of other excellent messaging systems such as RabbitMQ and MQTT. Where Kafka is being recognized is in the areas of high volume performance, clustering and reliability.

Like RabbitMQ and MQTT, Kafka messaging are defined as topics. Topics can be produced (published) and consumed (subscribed). Where Kafka differs is in the storage of messages. Kafka stores all produced topic messages up until a defined time out.

Node-Red is an open source visual programming tool that connects to Raspberry Pi hardware and it has web dashboards that can be used for Internet of Things presentations.

In this blog I would like to look at using Node-Red with Kafka for Internet of Things type of applications.

Getting Started

Kafka can be loaded on a variety of Unix platforms and Windows.  A Java installation is required for Kafka to run, and it can be installed on an Ubuntu system by:

apt-get install default-jdk

For Kafka downloads and installation instructions see: https://kafka.apache.org/quickstart. Once the software is installed and running there a number of command line utilities in the Kafka bin directory that allow you to do some testing.

To test writing messages to a topic called iot_test1, use the kafka-console-producer.sh  command and enter some data (use Control-C to exit):

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic iot_test1
11
22
33

To read back and listen for messages:

 bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic iot_test1 --from-beginning
11
22
33

The Kafka server is configured in the /config/server.properties  file. A couple of the things that I tweeked in this file were:

# advertised the Kafka server node ip
advertised.listeners=PLAINTEXT://192.168.0.116:9092
# allow topics to be deleted
delete.topic.enable=true

Node-Red

Node-Red is a web browser based visual programming tool, that allows users to create logic by “wiring” node blocks together.  Node-Red has a rich set of add-on components that includes things such as: Raspberry Pi hardware, Web Dash boards, email, Tweeter, SMS etc.

Node-Red has been pre-installed on Raspbian since 2015. For full installation instructions see:  https://nodered.org/#get-started

To add a Node-Red component select the “Palette Manager”, and in the Install tab search for kafka. I found that the node-red-contrib-kafka-manager component to be reliable (but there are others to try).

For my test example I wanted to create a dashboard input that could be adjusted. Then read back the data from the Kafka server and show the result in a gauge.

This logic uses:

  • Kafka Consumer Group – to read a topic(s) from a Kafka server
  • Dashboard Gauge – to show the value
  • Dashboard Slider – allows a user to select a numeric number
  • Kafka Producer – sends a topic message to the Kafka server

nodered_kafka Double-click on the Kafka nodes and in the ‘edit configuration’ dialog create and define a Kafka broker (or server). Also add the topic that you wish to read/write to.

kafka_consume

Double-click on the gauge and slider nodes and define a Dashboard group. Also adjust the labels, range and sizing to meet your requirements.

kafka_gauge

After the logic is complete hit the Deploy button to run the logic. The web dashboard is available at: http://your_node_red_ip:1880/ui.

kafka_phone

Final Comment

I found Node-Red and Kafka to be easy to use in a simple standalone environment. However when I tried to connect to a Cloud based Kafka service (https://www.cloudkarafka.com/) I quickly realized that there is a security component that needs to be defined in Node-Red. Depending on the cloud service that is used some serious testing will probably be required.

 

Mosquitto MQTT Server

There are a number of excellent open source MQTT (Message Queue Telemetry Transport) brokers or servers that are available. Eclipse Mosquitto is a very popular light weight server that can be run on lower end hardware such as Raspberry Pi modules.

In this blog I’ll look at setting up a Mosquitto broker and I’ll setup some connections to it using testing tools and then use JavaScript with Web Sockets to look at Mosquitto diagnostics.

Getting Started

An MQTT project will have the following components:

  • MQTT Broker. The Mosquitto Server that manages and distributes messages
  • MQTT Publisher. A source for the data, such as a Arduino or Raspberry Pi
  • MQTT Subscriber. An application that wants the data

There is a good selection of MQTT libraries and test utilities for MQTT publishers and subscribers.

To install the Mosquitto broker and test utilities on a Raspberry Pi or Debian system enter:

sudo apt-get update
sudo apt-get install mosquitto mosquitto-clients

For Windows and other systems see the Mosquitto download documentation.

The features of the mosquitto broker are defined by the mosquitto configuration file (/etc/mosquitto/mosquitto.conf). For simple testing you can leave the configuration as is, however if you need to any added features like bridging of MQTT brokers, adding diagnostics or setting up MQTT for web access then you’ll need to edit mosquitto.conf.

For my project I wanted to add diagnostic (with a 10 second update), and I wanted MQTT to also run on Websockets. To do this I edited the config file by:

sudo nano /etc/mosquitto/mosquitto.conf

then I added the lines:

listener 8080
protocol websockets
sys_interval 10

The websockets can be set to any port number (9001, 8000,8080 etc.). After the file has been modified you’ll need to restart the Mosquitto service:

sudo service mosquitto restart

Test Clients

To test that your Mosquitto MQTT broker is running you can open 2 terminal sessions and run the mosquitto_sub and mosquitto_pub utilities.

In the first session we start mosquitto_sub with it subscribing to a topic called mytopic1:

~$ mosquitto_sub -t mytopic1
value1
1234

In the second session we use mosquitto_pub and we can publish some values to the topic mytopic1:

~$ mosquitto_pub -t mytopic1 -m "value1"
~$ mosquitto_pub -t mytopic1 -m 1234

If everything is working correctly then the published values should appear in the subscribing session.

JavaScript, Websockets and  Mosquitto Diagnostics

We can create a simple web page to test out the Javascript, Websockets and Mosquitto Diagnostics.

The Mosquitto Diagnostics have a topic of : $SYS/broker/… Some of these diagnostics include:

$SYS/broker/bytes/received
$SYS/broker/bytes/sent
$SYS/broker/clients/connected
$SYS/broker/clients/maximum
$SYS/broker/clients/total
$SYS/broker/heap/current size
$SYS/broker/heap/maximum size
$SYS/broker/load/connections/+
$SYS/broker/load/bytes/received/+
$SYS/broker/load/bytes/sent/+
$SYS/broker/load/messages/received/+
$SYS/broker/load/messages/sent/+
$SYS/broker/load/publish/dropped/+
$SYS/broker/load/publish/received/+
$SYS/broker/load/publish/sent/+
$SYS/broker/load/sockets/+
$SYS/broker/messages/received
$SYS/broker/messages/sent
$SYS/broker/publish/messages/received
$SYS/broker/publish/messages/sent
$SYS/broker/retained messages/count
$SYS/broker/store/messages/count
$SYS/broker/store/messages/bytes
$SYS/broker/subscriptions/count
$SYS/broker/version

where  + = 1min, 5min or 15min averages

For a full description of the Mosquitto diagnostics see: https://mosquitto.org/man/mosquitto-8.html .

Next we can create a web page where we can subscribe to some of these diagnostic topics. An MQTT javascript library can be remotely called from:

https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.js

This library is very similiar to Python Paho MQTT library, and it supports both subscribe and publish functions.

<html>
<head>
<title> MQTT Subscribe Message</title>
</head>
<script src="https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.js" type="text/javascript"></script>


<script>
// Define some MQTT variables


function sub_mqtt_msg() {
// Send an MQTT message
  var mqtt_server = document.getElementById("mqtt_server").value;
  var mqtt_port = Number(document.getElementById("mqtt_port").value);


  client = new Paho.MQTT.Client(mqtt_server, mqtt_port,"");
  client.onMessageArrived = onMessageArrived;
  client.onMessageArrived = onMessageArrived;
  client.connect({onSuccess:onConnect});
  document.getElementById("submsg").innerHTML = "Trying to connect...<br>";

}
function onConnect() {
  document.getElementById("submsg").innerHTML = "New connection made...<br>";
  var mqtt_destname = document.getElementById("mqtt_destname").value;  
  client.subscribe(mqtt_destname);
  document.getElementById("submsg").innerHTML = "Subscribing to topic: " + mqtt_destname + " ...<br>";
}
function onMessageArrived(message) {
  var result = message.destinationName + " : " + message.payloadString + "<br>";
  document.getElementById("submsg").innerHTML += result;
}

</script>
<body>
<h1>MQTT Subscribe Test Page</h1>

Server Name: <input type="text" id="mqtt_server" value="192.168.0.121"><br><br>
Websocket: <input type="text" id="mqtt_port" value="8080"><br><br>
DestinationName: <input type="text" id="mqtt_destname" value="$SYS/broker/#"><br><br>

<button onclick="sub_mqtt_msg()">Subscript to MQTT</button>
<hr>
<h2>Subscribed Messages:</h2>
<div id=submsg></div>
</body>

</html>

By subscribing to $SYS/broker/# we can get see all the different diagnostic values.

java_ws_diag2

Summary

Mosquitto is an excellent MQTT broker that can be run on both high end servers and low end hardware like Raspberry Pi’s.

It is important to note that with most MQTT libraries, (i.e. Python Paho, JavaScript and Node Red) the default setting is to not retain only values. This can be a problem if a new MQTT subscribers starts because it will not have any data until new values are published. To ensure that old values are available set the RETAIN=YES option on the MQTT publish call.

In this blog I wanted to examine, the diagnostic features of the Mosquitto MQTT broker, however there are many more features and functions of MQTT communications that need to be examined. There are some excellent MQTT guides on the Internet. I tried to document some of my notes on MQTT with Javascript and Charts, and MQTT with Arduino and Node Red.

 

 

 

 

 

 

DynamoDB with Python and JavaScript

Amazon Web Services (AWS) is a probably the best known Cloud services offering on the Internet.  At first glance AWS is a quite intimidating, learning about all the different features of AWS could be a full time job. For this blog I wanted to introduce one component of AWS called DynamoDB.

DynamoDB is a NoSQL database that could be connected to sensor hub devices such as Raspberry Pi’s for Internet of Things (IoT) solutions.

aws_services

Getting Started with DynamoDB

The first step to getting started with AWS is to create an account. You can open a free “testing” account but unfortunately you’ll need to show that you’re serious by including a credit card number.

After creating an account you can sign into the AWS Management Console (https://aws.amazon.com/console/). With a single user account it is possible to run different services in different locations. It is important to note that not all services are available in all locations, so select a location that offers all the services that you need.

AWS_Region

After a region is selected go to the Services menu item and search for DynamoDB, next select “create table”. A DynamoDB table has a NoSQL database structure, so this means that all that is required is a table name and a primary key. A primary key allows rows of data to have a way to be connected together.

For my IoT table example I created a primary key called signalname. (If you plan to have a very large data set and you need extremely high speed response you might want to consider defining multiple keys, such as location or sensor type).

create_table

Once the table is created, you can view and modify data.

the_table

The “Items” tab is used to add, view and delete data. Note, the viewing of updates is not automatic, so you need to press the “refresh” button periodically.

mytag1

Users and Security

The AWS user security is quite detailed. To get started select your account name on the menu bar and “My Security Credentials“.  The Groups item allows you to create security classifications for activities such as admin, viewing, etc.

New users can be defined and added into security groups.

user_security

When a new users is created an Access Key ID and a Secret Access Key is generated. The Secret Access Key is only shown when the user is created so ensure that you save it, and/or download the CSV file.

newuser

After an Access key ID and Secret Access Key is created for a user, you can start  programming interfaces to DynamoDB.

DynamoDB and Python

The Python AWS library is called boto3 and it is installed  in Windows by :

pip install boto3

and in Linux by:

sudo pip install boto3

The AWS security can be either added directly into the Python (or JavaScript) application or it can reside on the node or PC that you are working on. Both methods have pro’s and con’s. To add the credentials locally, you’ll need to install the AWS command line tool (awscli) and then enter your specific information:

$ sudo pip install awscli
$ aws configure
AWS Access Key ID [None]: AKIAIDxxxxx
AWS Secret Access Key [None]: kzzeko8Fxxxxxxxxxxxxxxxxx
Default region name [None]: Central
Default output format [None]:

A Python example to write and read a row into DynamoDB with the credentials in the program could be:

import boto3
import time

thesec = int(time.time() % 60)

mysession = boto3.session.Session(
    aws_access_key_id= 'AKIAID7xxxxxxxxx',
    aws_secret_access_key='kzzeko8xxxxxxxxxxxxxxxx')
db = mysession.resource('dynamodb', region_name='ca-central-1')

table = db.Table('my_iot_burl1')

response = table.put_item(
   Item={
        'signal': 'mytag1',
        'tagdesc': 'test tag1',
        'tagvalue': thesec
    })

response = table.get_item(Key={'signal': 'mytag1'})
print(thesec)
print(response)
print(response['Item']['tagvalue'])

If the credentials are locally used then the code to write data would be:

import boto3

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('users01')

table.put_item(
   Item={
        'signal': 'mytag1',
        'tagdesc': 'test tag1',
        'tagvalue': 16
    }
)

The response comes back as JSON data. For this example the result would be:

16
{‘Item’: {‘tagvalue’: Decimal(’16’), ‘tagdesc’: ‘test tag1’, ‘signal’: ‘mytag1’}, ‘ResponseMetadata’: {‘RequestId’: ’66M7R4CVCACA4KNA24LM90KG0NVV4KQNSO5AEMVJF66Q9ASUAAJG’, ‘HTTPStatusCode’: 200, ‘HTTPHeaders’: {‘server’: ‘Server’, ‘date’: ‘Fri, 15 Feb 2019 16:29:48 GMT’, ‘content-type’: ‘application/x-amz-json-1.0’, ‘content-length’: ’84’, ‘connection’: ‘keep-alive’, ‘x-amzn-requestid’: ’66M7R4CVCACA4KNA24LM90KG0NVV4KQNSO5AEMVJF66Q9ASUAAJG’, ‘x-amz-crc32’: ‘220267538’}, ‘RetryAttempts’: 0}}
16

For large amounts of data being written to DynamoDB it is possible to use a batch writing function:

import boto3
import time

db = boto3.resource('dynamodb', region_name='ca-central-1')

table = db.Table('my_iot_burl1')

thesec = int(time.time() % 60)

with table.batch_writer() as batch:
    for i in range(1,11):
        batch.put_item(
            Item={
                'signal': 'mytag' + str(i),
                'tagdesc': 'test tag'+ str(i),
                'tagvalue': thesec,
                'status': 'NORMAL'
            }
        )

The AWS console can be used to verify that the data was written in.

Writing to DynamoDB will clear the existing row with a completely new set of data. If you only want to update a field in a row then the update_item call is used. In the example below only the sensor tagvalue is updated and all other fields are left unchanged.

import boto3

mysession = boto3.session.Session(
    aws_access_key_id= 'AKIAIDxxxxxxxxx',
    aws_secret_access_key='kzzekxxxxxxxxxxxxxxx'
)

db = mysession.resource('dynamodb', region_name='ca-central-1')

table = db.Table('my_iot_burl1')

# set the value for mytag1 to bet 29
table.update_item(
    Key={ 'signal': 'mytag1' },
    UpdateExpression='SET tagvalue = :newval1',
    ExpressionAttributeValues={
        ':newval1': 29
    }
)

Python “Scan the Data” Example

The boto3 library supports the ability to scan the data based on some filtering criteria. The filter syntax is: Attr(‘the field’).eq(‘thevalue’).

The following comparison operators are available:

eq | ne | le | lt | ge | gt | not_null | null | contains | not_contains | begins_with | in | between

Multiple statements can be created with | (OR) , & (AND) operators. Below is an example that scans the database for rows with an alarm status or with a tag value greater than 100.

scan_view

import boto3
from boto3.dynamodb.conditions import Key, Attr

db = boto3.resource('dynamodb', region_name='ca-central-1')

table = db.Table('my_iot_burl1')

response = table.scan(
    FilterExpression=Attr('status').eq('ALARM') | Attr('tagvalue').gt(100)
)

#print (response)
for i in response['Items']:
    print(i)
    print(i['signal'], "=", i['tagvalue'])

The results were:

{‘tagvalue’: Decimal(‘150’), ‘signal’: ‘mytag2’, ‘tagdesc’: ‘test tag2’, ‘status’: ‘NORMAL’}
mytag2 = 150
{‘tagvalue’: Decimal(‘999’), ‘signal’: ‘mytag1’, ‘tagdesc’: ‘test tag1’, ‘status’: ‘ALARM‘}
mytag1 = 999

JavaScript

For standard HTML/JavaScript projects the credentials are included in the source file. This obviously has some serious security issues, so it’s recommended that you define a low level user that can only do a small subset of functions.

Below is a simple example that can read our DynamoDB table and it can write a new row with an updated value.

<html>
<head>
<script src="https://sdk.amazonaws.com/js/aws-sdk-2.7.16.min.js"></script>

<script>
AWS.config.update({
  region: "ca-central-1",
  accessKeyId: "AKIAID7xxxxxxxxxxxx",
  secretAccessKey: "kzzeko8xxxxxxxxxxxxx"
});

var docClient = new AWS.DynamoDB.DocumentClient();

function createItem() {
  var d = new Date();
  var n = d.getSeconds();
    var params = {
        TableName :"my_iot_burl1",
        Item:{
      'signal': 'mytag4',
      'desc': 'test tag4',
      'info': {
        'value':n,
        'quality': "questionable"
        }
    }
    };
    docClient.put(params, function(err, data) {
        if (err) {
            document.getElementById('textarea').innerHTML = "Unable to add item: " + "\n" + JSON.stringify(err, undefined, 2);
        } else {
            document.getElementById('textarea').innerHTML = "PutItem succeeded: " + "\n" + JSON.stringify(params,undefined, 2);
        }
    });
}

function readItem() {
    var table = "my_iot_burl1";
    var signal = 'mytag4';
    var params = {
        TableName: table,
        Key:{'signal' : 'mytag4' }
    };
    docClient.get(params, function(err, data) {
        if (err) {
            document.getElementById('textarea2').innerHTML = "Unable to read item: " + "\n" + JSON.stringify(err, undefined, 2);
        } else {
            document.getElementById('textarea2').innerHTML = "GetItem succeeded: " + "\n" + JSON.stringify(data, undefined, 2);
        }
    });
}

</script>
</head>

<body>
<input id="createItem" type="button" value="Create Item" onclick="createItem();" />
<br><br>
<textarea readonly id= "textarea" style="width:400px; height:300px"></textarea>
<hr>
<input id="readItem" type="button" value="Read Item" onclick="readItem();" />
<br><br>
<textarea readonly id= "textarea2" style="width:400px; height:300px"></textarea>
</body>
</html>

aws_html

Summary

On first impressions I was pretty overwhelmed using Amazon Web Services. However after some time playing with it I found that it wasn’t unlike many other Web services that are available.

DynamoDB can be used to communicate with sensor values from a Raspberry Pi, but unfortunately there are no mainstream libraries for Arduino projects. AWS does offer an MQTT solution that can be used to update a DynamoDB table.

For small IoT projects I feel that AWS is overkill, but for larger projects I would definitely consider using it.

RabbitMQ for IoT

For Internet of Things (IoT) projects that are a lot of different ways that the sensors, devices and client interfaces can be connected together. For many projects using simple MQTT (Message Queue Telemetry Transport) is all that you need. However if you’re trying to merge and build IoT projects that use both MQTT and AMQP (Advanced Message Queue Protocol) or require a REST API then you should take a look at RabbitMQ.

RabbitMQ is an open source middleware solution that natively uses AMQP communications but it has a good selection of plug-ins to support features like: MQTT, MQTT Web Sockets, HTTP REST API and server-to-server communications.

rabbitmq_overview

In this blog we will setting up a RabbitMQ server, and we will look at some of the differences between MQTT and AMQP messaging. Finally an example of an Anduino MQTT message will be presented as both an MQTT and an AMQP item in a Node-Red dashboard.

Getting Started

RabbitMQ can be installed on Window, Linux, MacOS systems and there are also some cloud based offerings. For small systems lower end hardware like a Raspberry Pi can be used.  For complete RabbitMQ installation instructions see: https://www.rabbitmq.com/download.html . To install and run RabbitMQ on a Ubuntu system enter:

sudo apt-get update
sudo apt-get install rabbitmq-server
sudo service rabbitmq-server start

The next step is to add some plug-ins. For my project I loaded the MQTT and Web Administration plug-ins:

sudo rabbitmq-plugins enable rabbitmq_mqtt
sudo rabbitmq-plugins enable rabbitmq-management

The rabbitmqctl command line tool allows you to configure and review the RabbitMQ server. To add a user admin1,  with password admin1, that has config, write and read rights for management and administrator access, enter:

sudo rabbitmqctl add_user admin1 admin1
sudo rabbitmqctl set_permissions -p / admin1 ".*" ".*" ".*"
sudo rabbitmqctl set_user_tags admin1 management administrator

After you’ve defined an administrative user the RabbitMQ web management plug-in can be accessed by: http://ip_address:15672 .

Rabbit_webadmin

The RabbitMQ Web Management tool offers an overview of the present system load, connections, exchanges and queues. 

The RabbitMQ Web Management tool is excellent for small manual changes, if however you are looking a doing a large number of additions or changes then, rabbitmqadmin, the command line management tool can be used. This tool is  installed by:

# Get the cli and make it available to use.
wget http://127.0.0.1:15672/cli/rabbitmqadmin
sudo chmod +x rabbitmqadmin

Comparing MQTT and AMQP

It’s useful to comment about some of the differences between MQTT (Message Queue Telemetry Transport) and AMQP (Advanced Message Queueing Protocol) .

MQTT is a light weight publish-subscribe-based messaging protocol that works well with lower end hardware and limited bandwidth. For Arduino type applications where you only need to pass some sensor data MQTT is an excellent fit.

AMQP has more overhead than MQTT, because it is a more advanced protocol that includes message orientation, queuing, routing, reliability and security. Presently there are no mainstream AMQP Arduino libraries, but numerous programming options for Raspberry Pi, Linux, Windows and MacOS systems exist.  An AMQP IoT example would be to send sensor failures and alarms to dedicated maintenance and alarm queues.

MQTT and AMQP Queues

One of the biggest differences in queues is that MQTT queues are designed to show you the last available message, where as AMQP will store multiple messages in a queue.

A published MQTT message contains a message body, a retain flag and a quality of service (QoS) value.

An AMQP message can be published with added properties such as: time stamp, type of message and expiration information. AMQP messages also support the addition of custom header values. Below is a Python publish example that defines the message type to be “Pi Sensor”, and custom headers are include for status and alarm state.

#!/usr/bin/env python
import pika

node = "192.168.0.121"
user = "pete"
pwd = "pete"

# Connect to a remote AMQP server with a username/password
credentials = pika.PlainCredentials(user, pwd)
connection = pika.BlockingConnection(pika.ConnectionParameters(node,
        5672, '/', credentials))                                    
channel = connection.channel()

# Create a queue if it doesn't already exist
channel.queue_declare(queue='Rasp_1',durable=True)

# Define the properties and publish a message
props = pika.BasicProperties(
    headers= {'status': 'Good Quality',"alarm":"HI"},
    type ="Pi Sensor")
channel.basic_publish(exchange='',
    routing_key='Rasp_1',body='99.5', properties = props)

connection.close()

For this example the Rasp_1 queue can be examined using the Queue->Get Message option in the Web Management Interface. 

q_prop

RabbitMQ Messaging

There are a variety of ways that AMQP messages can be published and subscribed to. The simplest way is to create a queue and then messages can be published and subscribed to from that queue.

rabbitmq_layoutTo help with the distribution and filtering of messages AMQP supports a number of different exchange types. Messages in an exchange use bindings based on a routing key to link them to a queue. 

The main types of exchanges are: direct, fanout, headers and topic. An IoT example of a direct exchange would be if a group of Raspberry Pi sensor values are going into a “Rasp Pi Sensor” exchange. When the Rasp Pi publishes a sensor result to the exchange the message also includes a routing key to link the message to the correct queue.

direct

An IoT example of a fanout exchange would be with critical sensor failures. The sensor failure message is sent to Bill’s and Sam’s work or task queue and the All Maintenance point queue at the same time.

fanout

Connecting MQTT

After the MQTT plug-in is installed RabbitMQ can act like a standalone MQTT broker. MQTT data can also be made available through an AMQP subscription by binding the MQTT exchange to a RabbitMQ queue.

For an MQTT project any ESP8266 supported Arduino hardware can be used. There are a number of MQTT Arduino libraries that are available. For this project I used the PubSubClient that can be installed using the Arduino Library Manager.

As a test project I used at low cost MQ-2 Combustible Gas Sensor ($3) that measures a combination of LPG, Alcohol, Propane, Hydrogen, CO and even methane. Note to fully use this sensor some calibration is required. On the MQ-2 sensor the analog signal is connected to Arduino pin A0 and the analogRead(thePin) function is used to read the sensor value.

MPq2_Setup.png

Below is some example Arduino code required to read the MQ2 gas sensor and publish it to the RabbitMQ MQTT broker with a topic name of : mq2_mqtt.

/*
 Basic ESP8266 MQTT publish client example
*/
#include <ESP8266WiFi.h>
#include <PubSubClient.h>

// Update these with values suitable for your network.
const char* ssid = "your_ssid";
const char* password = "your_password";
const char* mqtt_server = "192.168.0.121"; 
const char* mqtt_user = "admin1";
const char* mqtt_pass= "admin1";

const int mq2pin = A0; //the MQ2 analog input pin

WiFiClient espClient;
PubSubClient client(espClient);

void setup_wifi() {
  // Connecting to a WiFi network
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}

void reconnect() {
  // Loop until we're reconnected
  Serial.println("In reconnect...");
  while (!client.connected()) {
    Serial.print("Attempting MQTT connection...");
    // Attempt to connect
    if (client.connect("Arduino_Gas", mqtt_user, mqtt_pass)) {
      Serial.println("connected");
    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
      delay(5000);
    }
  }
}

void setup() {
  Serial.begin(9600);
  setup_wifi();
  client.setServer(mqtt_server, 1883);
}

void loop() {
  char msg[8];
  if (!client.connected()) {
    reconnect();
  }

  sprintf(msg,"%i",analogRead(mq2pin));
  client.publish("mq2_mqtt", msg);
  Serial.print("MQ2 gas value:");
  Serial.println(msg);

  delay(5000);
}

Once the MQTT value is published any MQTT client can subscribe to it.  Below is a Python MQTT subscribe example.

# mqtt_sub.py - Python MQTT subscribe example 
#
import paho.mqtt.client as mqtt
 
def on_connect(client, userdata, flags, rc):
    print("Connected to broker")
 
def on_message(client, userdata, message):
    print ("Message received: "  + message.payload)

client = mqtt.Client()
client.username_pw_set("admin1", password='admin1')
client.connect("192.168.0.121", 1883) 

client.on_connect = on_connect       #attach function to callback
client.on_message = on_message       #attach function to callback

client.subscribe("mq2_mqtt") 
client.loop_forever()                 #start the loop

Read MQTT Messages Using AMQP

MQTT clients can subscribe to MQTT messages directly or it’s possible to configure RabbitMQ to have the MQTT data accessible as AMQP. The routing of MQTT messages to AMQP queues is done using the direct exchange method.

mqtt_2_amqp

To configure RabbitMQ to forward MQTT the following steps are done:

  1. Create a new RabbitMQ Queue – For an IoT project this would typically be a 1-to-1 mapping of the MQTT topic to a queue name.
  2. Create a binding between the MQTT Exchange and the Queue – by default all the MQTT topic go into the amq.topic exchange. For MQTT items in this exchange a binding, typically the MQTT topic name, is used as the routing key to the AMQP queue.

These steps can be done in a number of ways : manually, in the RabbitMQ config file, using the rabbitmqadmin command line tool or via a program. Because I was doing this for multiple signals I used rabbitmqadmin tool, and the syntax is:

./rabbitmqadmin declare queue name=mq2_amqp durable=true
./rabbitmqadmin declare binding source=amq.topic destination_type=queue destination=mq2_amqp routing_key=mq2_mqtt

The RabbitMQ Web Admin can be used to verify the exchange to queue binding.

wp_mqtt_2_amqp

 

The CLI tool can also be used to see if there are any values in the queue:

cli_get_queue

Node-Red Dashboard

Node-Red is a visual programming environment that allows users to create applications by dragging and dropping nodes on the screen. Logic flows are then created by connecting the different nodes together.

Node-Red has been preinstalled on Raspbian Jesse since the November 2015. Node-Red can also be installed on Windows, Linux and OSX.  To install and run Node-Red on your specific system see https://nodered.org/docs/getting-started/installation.

To install the AMQP components, select the Manage palette option from the right side of the menu bar. Then search for “AMQP” and install node-red-contrib-amqp. If your installation of Node-Red does not have dashboards installed, search for: node-red-dashboard.

nr_install_amqp

For our Node-Red MQTT and AMQP example we will use a mqtt and a amqp node from the input palette group, and two gauge nodes from the dashboard group.  The complete Node-Red logic for this is done in only 4 nodes!

nr_logic

Nodes are added by dragging and dropped them into the center Flow sheet. Logic is created by making connection wires between inputs and output of a node. After the logic is laid out, double click on each of the nodes to configure their specific properties. You will need to specify the MQTT and AMQP definitions of your RabbitMQ IP address,user rights, MQTT topic and AMQP queue name. You will also need to double click on the gauge nodes to configure the look-and-feel of the web dashboard.

After the logic is complete, hit the Deploy button on the right side of the menu bar to run the logic.  The Node-Red dashboard user interface is accessed by: http://ipaddress:1880/ui. 

For my project I used a number of different MQ sensors and inputs. Below is a picture of Node-Red web dashboard that we created with the same MQTT value being shown natively and as a AMQP queued value.

project

Final Comments

I found that RabbitMQ was easy to install and the Web Administration plug-in along with the rabbitmqadmin command line tool made it very easy to maintain. 

If you’re just looking to show sensor values then a basic MQTT broker might be all you need. However if you’re looking at some future applications like alarm, maintenance or task lists then AMQP exchanges and queue make RabbitMQ an interesting option.

More on RabbitMQ:

RabbitMQ REST API – remote interfacing (Javascript and Python examples)

RabbitMQ connections with Excel

MQTT and Javascript

MQTT (Message Queuing Telemetry Transport) is a  publish-subscribe-based messaging protocol that is used on many Internet of Things (IoT) projects. It works on top of the TCP/IP protocol and it is designed for connections with remote locations where a “small code footprint” is required or the network bandwidth is limited. The publish-subscribe messaging pattern requires a message broker.

MQTT_js_overview

There is support for MQTT on a large variety of programming languages and platforms. An Arduino or Raspberry Pi module can sent (or publish) I/O to a MQTT broker, and they can also receive (or subscribe) to data.

There are a number of MQTT brokers that can be used. One of the most popular ones is the Mosquitto MQTT broker, and it can be loaded on Windows, OSX and Linux systems. For this blog we will be using the Mosquitto test MQTT server. This Internet based server should not be used for real systems, but it is excellent for small short terms tests.

MQTT Web Sockets

The MQTT server has configurable MQTT and Web Socket ports. For a typical Raspberry Pi or Arduino connection, the default MQTT port 1883 would be used. In many Internet applications only certain ports are open, so for this reason a different MQTT Web Socket is used. This is configurable but ports like 80 or 8080 are typically used.MQTT_web_layout

Javascript Application

There are a number of MQTT javascript libraries that are available. One of the more popular ones is the Paho library that is available at:

https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.js

As a first example we will look at creating 2 pages. The first page will  publish a value, and the second page will subscribe to the data.

MQTT_js_pubsub

The publish code is:

<html>
<head>
<title> MQTT Publish Message</title>
</head>
<script src="https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.js" type="text/javascript"></script>

<script>
// Define some MQTT variables
var mqtt_server = "";
var mqtt_port = "";
var mqtt_destname = "";

function send_mqtt_msg() {
// Send an MQTT message
  mqtt_server = document.getElementById("mqtt_server").value;
  mqtt_port = Number(document.getElementById("mqtt_port").value);
 

  client = new Paho.MQTT.Client(mqtt_server, mqtt_port,"");
  client.connect({onSuccess:onConnect});
  document.getElementById("pubmsg").innerHTML = "Trying to connect...
";
}
function onConnect() {
  document.getElementById("pubmsg").innerHTML = "New connection made...
";
  var mqtt_destname = document.getElementById("mqtt_destname").value;
  var mqtt_msg = document.getElementById("mqtt_msg").value;   
  message = new Paho.MQTT.Message(mqtt_msg);
  message.destinationName = mqtt_destname;
  client.send(message);
  document.getElementById("pubmsg").innerHTML = "topic:" + mqtt_destname + " = " + mqtt_msg + " ...sent";
}  
// called when a message arrives
</script>
<body>
<h1>MQTT Publish Test Page</h1>

Server Name: <input type="text" id="mqtt_server" value="test.mosquitto.org"><br><br>
Websocket: <input type="text" id="mqtt_port" value="8080"><br><br>
DestinationName: <input type="text" id="mqtt_destname" value="my_IoT_value1"><br><br>
Message: <input type="text" id="mqtt_msg" value="test message"><br><br>

<button onclick="send_mqtt_msg()">Publish MQTT Message</button>
</body>
<hr>
<div id=pubmsg></div>
</html>

The subscribe code is:

<html>
<head>
<title> MQTT Subscribe Message</title>
</head>
<script src="https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.js" type="text/javascript"></script>
<script>

function sub_mqtt_msg() {
// Send an MQTT message
  var mqtt_server = document.getElementById("mqtt_server").value;
  var mqtt_port = Number(document.getElementById("mqtt_port").value);

  client = new Paho.MQTT.Client(mqtt_server, mqtt_port,"");
  client.onMessageArrived = onMessageArrived;
  client.onMessageArrived = onMessageArrived;
  client.connect({onSuccess:onConnect});
  document.getElementById("submsg").innerHTML = "Trying to connect...
";

}
function onConnect() {
  document.getElementById("submsg").innerHTML = "New connection made...
";
  var mqtt_destname = document.getElementById("mqtt_destname").value;  
  client.subscribe(mqtt_destname);
  document.getElementById("submsg").innerHTML = "Subscribing to topic: " + mqtt_destname + " ...
";
}
function onMessageArrived(message) {
  var result = message.destinationName + " : " + message.payloadString + "
";
  document.getElementById("submsg").innerHTML = result;
}

</script>
<body>
<h1>MQTT Subscribe Test Page</h1>

Server Name: <input type="text" id="mqtt_server" value="test.mosquitto.org"><br><br>
Websocket: <input type="text" id="mqtt_port" value="8080"><br><br>
DestinationName: <input type="text" id="mqtt_destname" value="my_IoT_value1"><br><br>

<button onclick="sub_mqtt_msg()">Subscript to MQTT</button>
<hr>
<h2>Subscribed Messages:</h2>
<div id=submsg></div>
</body>

</html>

Once you’ve got the basics down it’s possible to make some more advanced web interface pages.

Chart and Gauges

For your IoT projects there are a lot of Dash board options that are available. One of my favorites is Node-Red because it is totally free and standalone.

If you would like to create your own web interface there are a number of good javascript charting and gauge libraries available. For my examples I used Google charts with the Gauge chart . To simulate the inputs I used three of our MQTT Publish pages.

MQTT_js_sim

The code for the gauges page is :

<html>
<head>
<title>IoT - MQTT to JS</title>
</head>
<script src="https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.js" type="text/javascript"></script>
<script src="https://www.gstatic.com/charts/loader.js" type="text/javascript"></script>

<script>
// MQTT variables
var MQTTnames = ["my_IoT_value1","my_IoT_value2","my_IoT_value3"];
var MQTTvalues = [0,0,0];


// Define the Google gauge chart
      google.charts.load('current', {'packages':['gauge']});
      google.charts.setOnLoadCallback(drawChart);

      function drawChart() {

        var data = google.visualization.arrayToDataTable([
          ['Label', 'Value'], 
      [MQTTnames[0], MQTTvalues[0]],
      [MQTTnames[1], MQTTvalues[1]],
      [MQTTnames[2], MQTTvalues[2]],      
        ]);

        var options = {
          width: 800, height: 1000,
          redFrom: 90, redTo: 100,
          yellowFrom:75, yellowTo: 90,
          minorTicks: 5
        };

        var chart = new google.visualization.Gauge(document.getElementById('chart_div'));

        chart.draw(data, options);

        setInterval(function() {
      for (var i=0; i < MQTTnames.length; i++) {
      data.setValue(i, 1, MQTTvalues[i]);
      }
          chart.draw(data, options);

      }, 1000);
}

// Create a client instance
client = new Paho.MQTT.Client("test.mosquitto.org", 8080,"");
client.onMessageArrived = onMessageArrived;

// connect the client
client.connect({onSuccess:onConnect});

// called when the client connects
function onMessageArrived(message) {
  
  for (var i=0; i < MQTTnames.length; i++) {
  if (message.destinationName == MQTTnames[i]) {
    MQTTvalues[i] = Number(message.payloadString);
  }
  }
}
function onConnect() {
  // Once a connection has been made, make a subscription and send a message.
  for (var i=0; i < MQTTnames.length; i++) {
  client.subscribe(MQTTnames[i]);
  }
}
// called when a message arrives
</script>
<body>
<h1  style = 'font-size: xx-large'>IoT - MQTT to JavaScript</h1>
<div id="chart_div" style="width: 100%; height: 100%;"></div>
</body>
</html>

There are many other charting options that could be used. Below is an example using the Google Chart library with bars instead of gauges.

MQTT_js_bars.png

Final Comment

Using Javascript to interact with your IoT projects open up a lot of potential for adding functionality. I found that the Python version of the Paho MQTT library to have better documentation and perhaps some more functions, but at the end of the day I was able to do all the I wanted.