Arduino Yún for IoT

There are some great Internet of Things hardware platforms, such as Raspberry Pi’s, ESP-32 based Arduino modules, Particle and Beaglebone to name just a few. Each of these systems have their strengths and weakness. Some are strong on the hardware side, like the Arduino modules, and others excel on the programming side like the Raspberry Pi.  

The Yún is somewhat unique in that it’s a module with two processors, one that supports standard Arduino programming and a second processor that supports Linux and the OpenWrt wireless stack. The Yún ($59) has an Arduino Uno form factor and there are clones like the LinkIt Smart 7688 Duo ($18) in an Arduino Nano form factor.


In this blog I wanted to document some of key feature and functions that I worked through, namely:

  • moving file – scp and ftp
  • Python bridging to Arduino
  • uhttpd Web Server – with Python CGI
  • Yún REST API
  • MQTT
  • Yún mailbox

Yún Overview

The Yún, now in revision 2, seems to have been somewhat overlooked because of the all the low cost ESP-8266 and ESP32 based Arduino modules starting around $2.

Some the interesting features of the Yún include:

  • Arduino code can launch and get feedback from Linux apps
  • A bridge library allows Linux programs and Arduino to share data
  • Arduino libraries for:
    • Web client
    • Web server
    • Mailbox to Linux
  • Yún has a lightweight web server, uhttpd, that can be used to Python, PHP, Lua CGI web programs
  • Yún has a read/write REST API web service

Getting Started

The Arduino side of the Yún is available “out of the box” like any other Arduino module when you connect a USB cable into the module.

The Linux side however requires some configuration. Please see one of the guides on the Linux setup. Once your module is connected to your network, the Yun webserver can be used to configure features and add software components.


Another option for loading software is to use the opkg package manager from an SSH connection.


So for example to install nano, the simple text editor, enter:

 opkg install nano

Moving File and Working in OpenWrt Linux

The OpenWrt Linux isn’t loaded with with an X-windows environment and this means that you can not run idle or leafpad etc. to do easy editing of program files. Nano is a good command line text editor but it’s not the same as a windows based editor.

To move files between a PC and OpenWrt you have some options. The two that I like are:

  • scp – secure copy, this is built in both OpenWrt and Microsoft Window 10
  • ftp – file transfer protocol. Needs to be installed in OpenWrt.

There are a few ftp servers that could be installed in OpenWrt, vsftp is a lightweight option and it can be installed by:

opkg update
opkg install vsftpd

Once vsftpd is installed it needs to be enabled, this can be done from the command line or via the web interface.


Yún Bridge

The Yun Arduino bridge library allow variables to be passed between the Arduino code and a Python program. Below is an example that writes two random variables available, and it creates a “bridge1” variable that can be written from the Python side.

// Simple Yun Bridge Example

#include <Bridge.h>
// create a bridge variable to get remote data
char bridge_Value[10];

void setup()
    Bridge.begin();     // this launches /usr/bin/run-bride on Linino

void loop()
    // create 2 random bridge values, that are sourced from the Arduino
    Bridge.put("random1", String(random(1, 100)));
    Bridge.put("random2", String(random(1, 100)));
    // Called the bridge value "bridge1". This is the name used on the Python side
    Bridge.get("bridge1", bridge_Value, 6);

An example to read/write in Python:


import sys

sys.path.insert(0, '/usr/lib/python2.7/bridge/')

from bridgeclient import BridgeClient as bridgeclient
value = bridgeclient()

message = value.get("random1") # get a value from Arduino

print "Random1: ", message

value.put("bridge1","1111")  # set a value to Arduino

Yún uhttpd web server

The uhttp web server is used for Yun setup and configuration. This web server can also be used to for custom static pages and user web apps. To view/modify the web server settings:

nano /etc/config/uhttpd

Within this config file, you can enable custom web applications by defining an interpreter:

# List of extension->interpreter mappings.
# Files with an associated interpreter can
# be called outside of the CGI prefix and do
# not need to be executable.
# list interpreter ".php=/usr/bin/php-cgi"
# list interpreter ".cgi=/usr/bin/perl"
list interpreter ".py=/usr/bin/python

# Lua url prefix and handler script.
# Lua support is disabled if no prefix given.
# option lua_prefix /luci
# option lua_handler /usr/lib/lua/luci/sgi/uhttpd.lua

The default directory for user programs is: /www/cgi-bin

Python CGI – Get Values

To read the Arduino bridge values in a Python CGI program, add a file to the /www/cgi-bin directory. For my example I called the file :


import sys
import cgi

sys.path.insert(0, '/usr/lib/python2.7/bridge/')

from bridgeclient import BridgeClient as bridgeclient
value = bridgeclient()

print "Content-type:text/html\r\n\r\n"
print '<html>'
print '<head>'
print '<title>Python CGI Bridge</title>'
print '</head>'
print '<body>'
print '<h2>Python CGI Bridge</h2>'
print 'Random Value1: ' + value.get("random1")
print 'Random Value2: ' + value.get("random2")
print '</body>'
print '</html>'

To run the python script on the web page you will need to change the file right to executable:

chmod +x

You can debug and see output from the command line.

root@yun1:/www/cgi-bin# ./

<title>Python CGI Bridge</title>
<h2>Python CGI Bridge</h2>
Random Value1: 13
Random Value2: 24

If the output looks good, try the app from the Web page:


Python CGI – Put Values

There are a number of methods that can be used to send user input from a web page. A simple approach is to use a form. The form data can be read from the cgi.FieldStorage object, using a form.getvalue() call.


# Import modules for CGI handling
import cgi
import sys

sys.path.insert(0, '/usr/lib/python2.7/bridge/')
from bridgeclient import BridgeClient as bridgeclient
value = bridgeclient()

# Create instance of FieldStorage
form = cgi.FieldStorage()

# Get data from fields
bridge1 = form.getvalue('bridge1')
if bridge1 != "" :

print "Content-type:text/html\r\n\r\n"
print """
<title>Python CGI - Put Bridge Value</title>
<h2>Python CGI Form - Put Bridge Value</h2>
<form action = "/cgi-bin/" method = "post">
Enter BRIDGE1 value: <input type = "text" name = "bridge1"><br />

<input type = "submit" value = "Submit" />


The web page will call itself when the submit button is pressed.



The Yún REST API is a web service that allow remote users and web browsers to view and set bridge values. I found the REST API to be a good tool for testing my Web CGI and Python applications

To view all bridge variables enter: http://yourYun_IP/data/get


To get a specific bridge value enter:  http://yourYun_IP/data/get/my_bridge_value


To put a bridge value enter: http://yourYun_IP/data/put/my_bridge_value/value 


IoT Connections – MQTT

For Internet of Things (IoT) projects you need to pass data from the Arduino and some server. The communications and the server would be something like MQTT or Redis.

The Yún does not have direct access to the Wifi or ethernet port so the standard Arduino libraries for MQTT or Redis etc. will not work. An alternative approach is load the protocol’s command line client on the Linux side and then Arduino can shell out to the Linux tool.

For example to load the  Mosquitto MQTT command line tools:

opkg update
opkg install mosquitto-client

To test MQTT publishing to a topic (mytag1) with a message of 249 on remote client ( :

mosquitto_pub -h -u username -P password -t mytag1 -m 249

To remotely subscribe to

mosquitto_sub -h -u username -P password -t mytag1

An example of sending out four MQTT messages in Arduino:

Shell out to pass values to a MQTT command line pub
#include <Process.h>

void setup() {
  Bridge.begin();	// Initialize the Bridge
  Serial.begin(9600);	// Initialize the Serial

  // Wait until a Serial Monitor is connected.
  while (!SerialUSB);

void loop() {
  Process p;
  String thecmd;
  String strval = String( random(0,100));
  // create a string with host, username and password, -t is for the topic
  String theparams = "-h -u pete -P pete -t ";
  int numtopics = 4;
  String topics[4] = {"tag1","tag2","tag3","tag4"};

  for (int i=0; i < numtopics ; i++) {
    strval = String( (i*100) + random(1,99)); // create a random value - 0,100+,200+,300+
    thecmd = "mosquitto_pub " + theparams + topics[i] + " -m " + strval;
    // do nothing until the process finishes, so you get the whole output:
    while (p.running());
  // Wait 5 seconds and repeat
  delay(5000);  // wait 5 seconds before you do it again

It is also possible to create some bridge variables and pass them to a Python program that could do the MQTT communications.

Yún Mailbox

At present the mailbox only works in one direction, and this into the Arduino. Both the REST web interface and Python have writing functionality but no read capabilities (on the forums this has been identified, so a future revision may add this).

To write using REST:


To write using Python:

import sys
sys.path.insert(0, '/usr/lib/python2.7/bridge')
from bridgeclient import BridgeClient
client = BridgeClient()

The Arduino code to read message:

// Mailbox Read Example
#include <Mailbox.h>

void setup() {

  // Initialize Bridge and Mailbox

  // Initialize Serial

  // Wait until a Serial Monitor is connected.
  while (!SerialUSB);
  SerialUSB.println("Mailbox Read Message\n");

void loop() {
  String message;

  // if there is a message in the Mailbox
  if (Mailbox.messageAvailable()) {
    // read all the messages present in the queue
    while (Mailbox.messageAvailable()) {

    SerialUSB.println("Waiting 10 seconds before checking the Mailbox again");
  // wait 30 seconds

I’m not totally sure when I’d use a mailbox. The mailbox is a generic message and it’s not queued (only 1 message) so I think that using the standard bridging of values with get/put is more useful.

Final Comments

The Yun is missing a lot of networking libraries that are available with the ESP-8266 and ESP32 family of Arduino modules. In some cases like with MQTT that was a good Linux command line tool that could be used, but in cases like where you want to connect to a Redis server, you might have some challenges.

I really liked Yun’s built in uhttpd web server, this is far superior to the ESP-8266 Arduino web server library calls.

I found that putting logic into a combination of Python and Arduino could be a little confusing, but for projects with a lot of text or json data using Python would be a real plus. Also for projects where multitasking is required use the Linux process would be ideal.


RabbitMQ is an open source middleware message solution that natively uses AMQP communications but it has a good selection of plug-ins to support features like: MQTT, MQTT Web Sockets, HTTP REST API and server-to-server communications.


RabbitMQ has a very good selection of interfacing clients for most of the major programming languages. For applications where there is no custom programming library the REST API can be used.

Applications where the REST API could be used might include:

  • Arduino IoT projects
  • Web Clients (not using NodeJS)
    • For simple reading and writing of messages
    • For monitoring of RabbitMQ loading
  • AppInventor Android apps

Unfortunately I didn’t find the RabbitMQ REST API to be 100% straightforward, so in this blog I could like to document some of my notes and testing.

What is REST API

Representational State Transfer (REST) is a software style for creating Web services. A client issues an HTTP request to a server with a GET/POST/PUT or DELETE method and the server responds with a JSON (Javascript Object Notation) formatted message.


The most common tool to test REST API is curl. Curl is supported on all major OS’s. The curl syntax will vary slightly between Windows and Linux, especially in how the systems handle strings.

RabbitMQ Setup

RabbitMQ can be installed on Window, Linux, MacOS systems and there are also some cloud based offerings. For small systems lower end hardware like a Raspberry Pi can be used.  For complete RabbitMQ installation instructions see: . To install and run RabbitMQ on a Ubuntu system enter:

sudo apt-get update
sudo apt-get install rabbitmq-server
sudo service rabbitmq-server start

The next step is to add some plug-ins. For the Web Administration and REST API plug-ins:

sudo rabbitmq-plugins enable rabbitmq-management

The rabbitmqctl command line tool allows you to configure and review the RabbitMQ server. To add a user pete,  with password pete, that has config, write and read rights for management and administrator access, enter:

sudo rabbitmqctl add_user pete pete
sudo rabbitmqctl set_permissions -p / pete ".*" ".*" ".*"
sudo rabbitmqctl set_user_tags pete management administrator

After you’ve defined an administrative user the RabbitMQ web management plug-in can be accessed by: http://ip_address:15672 .


Once the Web Management component is installed it is much easier to test and play with REST API calls.


For a complete set of RabbitMQ REST API calls see: .

Typically you would use the RabbitMQ tools to create and exchanges and queues but it is possible to do this via the REST API interface.

Below are some examples using user: pete and a password of pete . Note: %2f is used for default (blank name) virtual host.

Create an exchange: webex

curl -i -u pete:pete -H “content-type:application/json” -X PUT -d”{“”type””:””direct””,””auto_delete””:false,””durable””:true,””internal””:false,””arguments””:{}}”

Create a Queue: webq1

curl -i -u pete:pete -H “content-type:application/json” -X PUT -d”{“”auto_delete””:false,””durable””:true,””arguments””:{}}”

Create Binding: Bind queue webq1 to exchange webex with a routing key of webq1

curl -i -u pete:pete -H “content-type:application/json” -X POST -d”{“”routing_key””:””webq1“”,””arguments””:{}}”

Comments: you need to bind a queue to an exchange if you wish to write to it. The routing key does the binding. The key can be any name but using the queue name makes it simple. A queue can also have binding to multiple exchanges.

Get a message: from queue webq1

curl -i -u pete:pete -H “content-type:application/json” -X POST -d”{“”count””:5,””requeue””:true,””encoding””:””auto””,””truncate””:50000}”

Comments: A queue can have multiple messages. A requeue=false, will remove messages that are already read.

Post a message (44.4) : to exchange webex with routing key webq1

curl -i -u pete:pete -H “content-type:application/json” -X POST -d”{“”properties””:{},””routing_key””:””webq1“”,””payload””:””44.4“”,””payload_encoding””:””string””}”


Javascript REST API Client

I had some Web browser issues when my test PC and my RabbitMQ server were in different domains. Web Browsers like Chrome and  Firefox will complain about a CORS policy problem.

Cross-origin resource sharing (CORS) is a mechanism that allows restricted resources on a web page to be requested from another domain outside the domain from which the first resource was served.

If you are crossing domains and you want to do some quick testing Microsoft Explorer will work.

Javascript GET a Queue Message

I used the AJAX XMLHttpRequest object to do my GET and POST requests.

<!DOCTYPE html>
<h2>RabbitMQ REST API - Get Queue Message</h2>

URL/API Call: <input type="text" id="theurl" value=""  size = "50" /> <br><br>
Queue Name:   <input type="text" id="qname" value="webex"/> <br><br>
Requeue: <input type="text" id="rq" value="true" /> (false = remove old values)<br><br>
Message Count: <input type="text" id="mcount" value="5"/> <br><br>
<button type="button" onclick="sendoutput()">Get Message(s)</button>

<p id="demo"></p>

function sendoutput() {
	// RabbitMQ username and password
	var username = "pete";
	var password = "pete";
	var xhttp = new XMLHttpRequest();
    var fullurl = document.getElementById("theurl").value + document.getElementById("qname").value + "//get";"POST", fullurl, true,username,password);
	// change to your Auth username and password


	var rq = document.getElementById("rq").value;
	var mcount = document.getElementById("mcount").value;
	var params = JSON.stringify({"count":mcount,"requeue":rq,"encoding":"auto","truncate":"50000"});
	// Define a handler function when the response comes back
	xhttp.onreadystatechange = function() {
		// Get the response and put on it the screen
		if (this.readyState == 4 ) {	
			document.getElementById("demo").innerHTML = "Result=" +xhttp.responseText;




Javascript POST an Exchange Message

Unfortunately you can not send a message directly to a queue, instead you need to send a message to an exchange with routing key. This routing key is linked to the queue that you wish to connect to.

<!DOCTYPE html>
<h2>RabbitMQ REST API - An Exchange Message</h2>

URL/API Call: <input type="text" id="theurl" value=""  size = "50" /> <br><br>
Exchange Name:   <input type="text" id="exname" value="webex"/> <br><br>
Routing Key:   <input type="text" id="rkey" value="webq1"/> <br><br>
Value to Send:   <input type="text" id="theval" value="11.1"/> <br><br>

<button type="button" onclick="sendoutput()">Send Output</button>

<p id="demo"></p>

function sendoutput() {
	// RabbitMQ username and password
	var username = "pete"; 
	var password = "pete";  
	var xhttp = new XMLHttpRequest();
	var fullurl = document.getElementById("theurl").value + document.getElementById("exname").value + "//publish";"POST", fullurl, true,username,password);

	var rkey = document.getElementById("r").value
	var theval = document.getElementById("theval").value
	var params = JSON.stringify({"properties":{},"routing_key":rkey,"payload":theval,"payload_encoding":"string"});
	// Define a handler function when the response comes back
	xhttp.onreadystatechange = function() {
		// Get the response and put on it the screen
		if (this.readyState == 4 ) {	
			document.getElementById("demo").innerHTML = "Result=" +xhttp.responseText;


Python – get message(s) from a queue

The requests library is extremely simple to use for REST interfacing. It is installed by:

pip install requests

# importing the requests library
import requests

# defining the api-endpoint

headers = {'content-type': 'application/json'}
# data to be sent to api
pdata = {'count':'5','requeue':'true','encoding':'auto','truncate':'50000'}

# sending post request and saving response as response object
r = = API_ENDPOINT ,auth=('pete', 'pete'), json = pdata, headers=headers)

# extracting response text
pastebin_url = r.text
print("Response :%s"%pastebin_url)

Python – Write a message to an exchange

# importing the requests library
import requests
import json

# defining the api-endpoint

# your source code here

headers = {'content-type': 'application/json'}
# data to be sent to api

pdata = {'properties':{},'routing_key':'webq1','payload':'90.4','payload_encoding':'string'}

# sending post request and saving response as response object
r = = API_ENDPOINT ,auth=('pete', 'pete'), json = pdata, headers=headers)

# extracting response text
pastebin_url = r.text
print("Response :%s"%pastebin_url)

Final Comments

The RabbitMQ REST interface is not designed for high throughput, but it is an excellent way to connect devices that have no native API support.

For my next round of testing I’d like to work on an Arduino and a AppInventor Android app.

Apache Kafka with Node-Red

Apache Kafka is a distributed streaming and messaging system. There are a number of other excellent messaging systems such as RabbitMQ and MQTT. Where Kafka is being recognized is in the areas of high volume performance, clustering and reliability.

Like RabbitMQ and MQTT, Kafka messaging are defined as topics. Topics can be produced (published) and consumed (subscribed). Where Kafka differs is in the storage of messages. Kafka stores all produced topic messages up until a defined time out.

Node-Red is an open source visual programming tool that connects to Raspberry Pi hardware and it has web dashboards that can be used for Internet of Things presentations.

In this blog I would like to look at using Node-Red with Kafka for Internet of Things type of applications.

Getting Started

Kafka can be loaded on a variety of Unix platforms and Windows.  A Java installation is required for Kafka to run, and it can be installed on an Ubuntu system by:

apt-get install default-jdk

For Kafka downloads and installation instructions see: Once the software is installed and running there a number of command line utilities in the Kafka bin directory that allow you to do some testing.

To test writing messages to a topic called iot_test1, use the  command and enter some data (use Control-C to exit):

bin/ --broker-list localhost:9092 --topic iot_test1

To read back and listen for messages:

 bin/ --bootstrap-server localhost:9092 --topic iot_test1 --from-beginning

The Kafka server is configured in the /config/  file. A couple of the things that I tweeked in this file were:

# advertised the Kafka server node ip
# allow topics to be deleted


Node-Red is a web browser based visual programming tool, that allows users to create logic by “wiring” node blocks together.  Node-Red has a rich set of add-on components that includes things such as: Raspberry Pi hardware, Web Dash boards, email, Tweeter, SMS etc.

Node-Red has been pre-installed on Raspbian since 2015. For full installation instructions see:

To add a Node-Red component select the “Palette Manager”, and in the Install tab search for kafka. I found that the node-red-contrib-kafka-manager component to be reliable (but there are others to try).

For my test example I wanted to create a dashboard input that could be adjusted. Then read back the data from the Kafka server and show the result in a gauge.

This logic uses:

  • Kafka Consumer Group – to read a topic(s) from a Kafka server
  • Dashboard Gauge – to show the value
  • Dashboard Slider – allows a user to select a numeric number
  • Kafka Producer – sends a topic message to the Kafka server

nodered_kafka Double-click on the Kafka nodes and in the ‘edit configuration’ dialog create and define a Kafka broker (or server). Also add the topic that you wish to read/write to.


Double-click on the gauge and slider nodes and define a Dashboard group. Also adjust the labels, range and sizing to meet your requirements.


After the logic is complete hit the Deploy button to run the logic. The web dashboard is available at: http://your_node_red_ip:1880/ui.


Final Comment

I found Node-Red and Kafka to be easy to use in a simple standalone environment. However when I tried to connect to a Cloud based Kafka service ( I quickly realized that there is a security component that needs to be defined in Node-Red. Depending on the cloud service that is used some serious testing will probably be required.


Small Standalone Databases for IoT

For many IoT projects you want to historically save your data, however if you’re only saving a small amount of data using a database server such as MySQL can be overkill.

I wanted to come up with a way to save my data and be able to port it between projects and hardware.  My goal was to use Python because it has the best Raspberry Pi support. I looked at some options like:

  • CSV files – simple but filtering and sorting isn’t so easy
  • dBase files – good “old school” approach, however it’s no longer support in Excel and MS Access so viewing the data offline is a little ugly
  • Python Pickle and PickleDB – great for saving game scores but no real sorting
  • SQLite – excellent light weight database

For my light weight Python applications I was really happy with TinyDB. TinyDB offers:

  • a JSON storage interface, so you could easily migrate to MongoDB (or equivalent)
  • pure Python with no dependencies,  (so it runs on all OS’s and versions of Python)
  • supports queries and searches

Getting Started with TinyDB

To install TinyDB :

pip install tinydb

TinyDB is standalone so there are no server components. Your Python code talks directly to one of more JSON files, or you can also create multiple tables in one JSON.

Below is a Python example that create a JSON file with two tables.

# Use TinyDB to create a JSON file with 2 tables
from tinydb import TinyDB, Query

db = TinyDB('db2.json')
db.purge_tables() # Clear out the file's old data

table = db.table('Moisture')


table2 = db.table('Light')



import json

with open('db2.json') as f:
data = json.load(f)

print("JSON file with 2 tables\n")
print(json.dumps(data, indent = 4, sort_keys=True))

The JSON file will have a format like:


A Test Example

For a test example I wanted to simulate a month’s worth of 5 minute data. Then I would try some queries to show the data.

# Create a month's worth of 10 minute simulated JSON data
# Use a format of:
#    "Pi_1": {
#        "1": {
#            "date": "2019-04-01",
#            "time": "01:00",
#            "moisture": 577,
#            "light": 30
#        }

from tinydb import TinyDB, Query
import random

db = TinyDB('/writing/blog/tinydb/db1.json')
table = db.table('Pi_1')

for iday in range(1,31):
    print("day: ",iday)
    for ihour in range(0,24):
        print("\thour: ", ihour)
        for imin in range(0,56,5):
            thedate = "2019-04-{:02}".format(iday)
            thetime = "{:02}:{:02}".format(ihour, imin)
            table.insert({"date": thedate,"time": thetime ,"moisture": random.randint(200,600),"light":  random.randint(20,100)})

print ("Finished updating JSON file\n\n")

Once the JSON data file was created I used a simple Python Tkinter application to query the data by day. My goal was to see how easy it was to query the data and return results. Below is a my TinyDB Tkinter viewing application and code. The bottom slider is used to select the day in the month.


# Use a Tkinter 
# Create a dialog that uses TinyDB to query 1 day of data
# from a JSON file
from tinydb import TinyDB, Query
import tinydb 
import tkinter as tk

db = TinyDB('db1.json')

# get a selected days data
def sel():
   thedate = "2019-04-{:02}".format(int(var.get()))
   table = db.table('Pi_1') 
   data1 =['date'] == thedate)
   if data_list.size() > 0:
       data_list.delete(1,data_list.size() -1 )
   for items in data1:
       data_list.insert(data_list.size()+ 1,
                        items['date'] + " " + items['time'] +
                        " -  Moisture : " + str(items['moisture']) +
                        " -  Light : " + str(items['light']) )

root = tk.Tk()
root.title("DBtiny - Daily Data")

var = tk.DoubleVar()
data_list = tk.Listbox(root,height=20, width= 60)
day_scale = tk.Scale( root,variable = var, from_=1,to=30,orient='horizontal',length=300 ).grid(row=2,column=1)

button1 = tk.Button(root, text="Get Daily Values", command=sel).grid(row=3,column=1)



TinyDB is a good solution for small IoT projects where I’m using Python and I’m not collecting a lot of data.

My next step would be to look at presenting the JSON data in Node-Red.


Mosquitto MQTT Server

There are a number of excellent open source MQTT (Message Queue Telemetry Transport) brokers or servers that are available. Eclipse Mosquitto is a very popular light weight server that can be run on lower end hardware such as Raspberry Pi modules.

In this blog I’ll look at setting up a Mosquitto broker and I’ll setup some connections to it using testing tools and then use JavaScript with Web Sockets to look at Mosquitto diagnostics.

Getting Started

An MQTT project will have the following components:

  • MQTT Broker. The Mosquitto Server that manages and distributes messages
  • MQTT Publisher. A source for the data, such as a Arduino or Raspberry Pi
  • MQTT Subscriber. An application that wants the data

There is a good selection of MQTT libraries and test utilities for MQTT publishers and subscribers.

To install the Mosquitto broker and test utilities on a Raspberry Pi or Debian system enter:

sudo apt-get update
sudo apt-get install mosquitto mosquitto-clients

For Windows and other systems see the Mosquitto download documentation.

The features of the mosquitto broker are defined by the mosquitto configuration file (/etc/mosquitto/mosquitto.conf). For simple testing you can leave the configuration as is, however if you need to any added features like bridging of MQTT brokers, adding diagnostics or setting up MQTT for web access then you’ll need to edit mosquitto.conf.

For my project I wanted to add diagnostic (with a 10 second update), and I wanted MQTT to also run on Websockets. To do this I edited the config file by:

sudo nano /etc/mosquitto/mosquitto.conf

then I added the lines:

listener 8080
protocol websockets
sys_interval 10

The websockets can be set to any port number (9001, 8000,8080 etc.). After the file has been modified you’ll need to restart the Mosquitto service:

sudo service mosquitto restart

Test Clients

To test that your Mosquitto MQTT broker is running you can open 2 terminal sessions and run the mosquitto_sub and mosquitto_pub utilities.

In the first session we start mosquitto_sub with it subscribing to a topic called mytopic1:

~$ mosquitto_sub -t mytopic1

In the second session we use mosquitto_pub and we can publish some values to the topic mytopic1:

~$ mosquitto_pub -t mytopic1 -m "value1"
~$ mosquitto_pub -t mytopic1 -m 1234

If everything is working correctly then the published values should appear in the subscribing session.

JavaScript, Websockets and  Mosquitto Diagnostics

We can create a simple web page to test out the Javascript, Websockets and Mosquitto Diagnostics.

The Mosquitto Diagnostics have a topic of : $SYS/broker/… Some of these diagnostics include:

$SYS/broker/heap/current size
$SYS/broker/heap/maximum size
$SYS/broker/retained messages/count

where  + = 1min, 5min or 15min averages

For a full description of the Mosquitto diagnostics see: .

Next we can create a web page where we can subscribe to some of these diagnostic topics. An MQTT javascript library can be remotely called from:

This library is very similiar to Python Paho MQTT library, and it supports both subscribe and publish functions.

<title> MQTT Subscribe Message</title>
<script src="" type="text/javascript"></script>

// Define some MQTT variables

function sub_mqtt_msg() {
// Send an MQTT message
  var mqtt_server = document.getElementById("mqtt_server").value;
  var mqtt_port = Number(document.getElementById("mqtt_port").value);

  client = new Paho.MQTT.Client(mqtt_server, mqtt_port,"");
  client.onMessageArrived = onMessageArrived;
  client.onMessageArrived = onMessageArrived;
  document.getElementById("submsg").innerHTML = "Trying to connect...<br>";

function onConnect() {
  document.getElementById("submsg").innerHTML = "New connection made...<br>";
  var mqtt_destname = document.getElementById("mqtt_destname").value;  
  document.getElementById("submsg").innerHTML = "Subscribing to topic: " + mqtt_destname + " ...<br>";
function onMessageArrived(message) {
  var result = message.destinationName + " : " + message.payloadString + "<br>";
  document.getElementById("submsg").innerHTML += result;

<h1>MQTT Subscribe Test Page</h1>

Server Name: <input type="text" id="mqtt_server" value=""><br><br>
Websocket: <input type="text" id="mqtt_port" value="8080"><br><br>
DestinationName: <input type="text" id="mqtt_destname" value="$SYS/broker/#"><br><br>

<button onclick="sub_mqtt_msg()">Subscript to MQTT</button>
<h2>Subscribed Messages:</h2>
<div id=submsg></div>


By subscribing to $SYS/broker/# we can get see all the different diagnostic values.



Mosquitto is an excellent MQTT broker that can be run on both high end servers and low end hardware like Raspberry Pi’s.

It is important to note that with most MQTT libraries, (i.e. Python Paho, JavaScript and Node Red) the default setting is to not retain only values. This can be a problem if a new MQTT subscribers starts because it will not have any data until new values are published. To ensure that old values are available set the RETAIN=YES option on the MQTT publish call.

In this blog I wanted to examine, the diagnostic features of the Mosquitto MQTT broker, however there are many more features and functions of MQTT communications that need to be examined. There are some excellent MQTT guides on the Internet. I tried to document some of my notes on MQTT with Javascript and Charts, and MQTT with Arduino and Node Red.







DynamoDB with Python and JavaScript

Amazon Web Services (AWS) is a probably the best known Cloud services offering on the Internet.  At first glance AWS is a quite intimidating, learning about all the different features of AWS could be a full time job. For this blog I wanted to introduce one component of AWS called DynamoDB.

DynamoDB is a NoSQL database that could be connected to sensor hub devices such as Raspberry Pi’s for Internet of Things (IoT) solutions.


Getting Started with DynamoDB

The first step to getting started with AWS is to create an account. You can open a free “testing” account but unfortunately you’ll need to show that you’re serious by including a credit card number.

After creating an account you can sign into the AWS Management Console ( With a single user account it is possible to run different services in different locations. It is important to note that not all services are available in all locations, so select a location that offers all the services that you need.


After a region is selected go to the Services menu item and search for DynamoDB, next select “create table”. A DynamoDB table has a NoSQL database structure, so this means that all that is required is a table name and a primary key. A primary key allows rows of data to have a way to be connected together.

For my IoT table example I created a primary key called signalname. (If you plan to have a very large data set and you need extremely high speed response you might want to consider defining multiple keys, such as location or sensor type).


Once the table is created, you can view and modify data.


The “Items” tab is used to add, view and delete data. Note, the viewing of updates is not automatic, so you need to press the “refresh” button periodically.


Users and Security

The AWS user security is quite detailed. To get started select your account name on the menu bar and “My Security Credentials“.  The Groups item allows you to create security classifications for activities such as admin, viewing, etc.

New users can be defined and added into security groups.


When a new users is created an Access Key ID and a Secret Access Key is generated. The Secret Access Key is only shown when the user is created so ensure that you save it, and/or download the CSV file.


After an Access key ID and Secret Access Key is created for a user, you can start  programming interfaces to DynamoDB.

DynamoDB and Python

The Python AWS library is called boto3 and it is installed  in Windows by :

pip install boto3

and in Linux by:

sudo pip install boto3

The AWS security can be either added directly into the Python (or JavaScript) application or it can reside on the node or PC that you are working on. Both methods have pro’s and con’s. To add the credentials locally, you’ll need to install the AWS command line tool (awscli) and then enter your specific information:

$ sudo pip install awscli
$ aws configure
AWS Access Key ID [None]: AKIAIDxxxxx
AWS Secret Access Key [None]: kzzeko8Fxxxxxxxxxxxxxxxxx
Default region name [None]: Central
Default output format [None]:

A Python example to write and read a row into DynamoDB with the credentials in the program could be:

import boto3
import time

thesec = int(time.time() % 60)

mysession = boto3.session.Session(
    aws_access_key_id= 'AKIAID7xxxxxxxxx',
db = mysession.resource('dynamodb', region_name='ca-central-1')

table = db.Table('my_iot_burl1')

response = table.put_item(
        'signal': 'mytag1',
        'tagdesc': 'test tag1',
        'tagvalue': thesec

response = table.get_item(Key={'signal': 'mytag1'})

If the credentials are locally used then the code to write data would be:

import boto3

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('users01')

        'signal': 'mytag1',
        'tagdesc': 'test tag1',
        'tagvalue': 16

The response comes back as JSON data. For this example the result would be:

{‘Item’: {‘tagvalue’: Decimal(’16’), ‘tagdesc’: ‘test tag1’, ‘signal’: ‘mytag1’}, ‘ResponseMetadata’: {‘RequestId’: ’66M7R4CVCACA4KNA24LM90KG0NVV4KQNSO5AEMVJF66Q9ASUAAJG’, ‘HTTPStatusCode’: 200, ‘HTTPHeaders’: {‘server’: ‘Server’, ‘date’: ‘Fri, 15 Feb 2019 16:29:48 GMT’, ‘content-type’: ‘application/x-amz-json-1.0’, ‘content-length’: ’84’, ‘connection’: ‘keep-alive’, ‘x-amzn-requestid’: ’66M7R4CVCACA4KNA24LM90KG0NVV4KQNSO5AEMVJF66Q9ASUAAJG’, ‘x-amz-crc32’: ‘220267538’}, ‘RetryAttempts’: 0}}

For large amounts of data being written to DynamoDB it is possible to use a batch writing function:

import boto3
import time

db = boto3.resource('dynamodb', region_name='ca-central-1')

table = db.Table('my_iot_burl1')

thesec = int(time.time() % 60)

with table.batch_writer() as batch:
    for i in range(1,11):
                'signal': 'mytag' + str(i),
                'tagdesc': 'test tag'+ str(i),
                'tagvalue': thesec,
                'status': 'NORMAL'

The AWS console can be used to verify that the data was written in.

Writing to DynamoDB will clear the existing row with a completely new set of data. If you only want to update a field in a row then the update_item call is used. In the example below only the sensor tagvalue is updated and all other fields are left unchanged.

import boto3

mysession = boto3.session.Session(
    aws_access_key_id= 'AKIAIDxxxxxxxxx',

db = mysession.resource('dynamodb', region_name='ca-central-1')

table = db.Table('my_iot_burl1')

# set the value for mytag1 to bet 29
    Key={ 'signal': 'mytag1' },
    UpdateExpression='SET tagvalue = :newval1',
        ':newval1': 29

Python “Scan the Data” Example

The boto3 library supports the ability to scan the data based on some filtering criteria. The filter syntax is: Attr(‘the field’).eq(‘thevalue’).

The following comparison operators are available:

eq | ne | le | lt | ge | gt | not_null | null | contains | not_contains | begins_with | in | between

Multiple statements can be created with | (OR) , & (AND) operators. Below is an example that scans the database for rows with an alarm status or with a tag value greater than 100.


import boto3
from boto3.dynamodb.conditions import Key, Attr

db = boto3.resource('dynamodb', region_name='ca-central-1')

table = db.Table('my_iot_burl1')

response = table.scan(
    FilterExpression=Attr('status').eq('ALARM') | Attr('tagvalue').gt(100)

#print (response)
for i in response['Items']:
    print(i['signal'], "=", i['tagvalue'])

The results were:

{‘tagvalue’: Decimal(‘150’), ‘signal’: ‘mytag2’, ‘tagdesc’: ‘test tag2’, ‘status’: ‘NORMAL’}
mytag2 = 150
{‘tagvalue’: Decimal(‘999’), ‘signal’: ‘mytag1’, ‘tagdesc’: ‘test tag1’, ‘status’: ‘ALARM‘}
mytag1 = 999


For standard HTML/JavaScript projects the credentials are included in the source file. This obviously has some serious security issues, so it’s recommended that you define a low level user that can only do a small subset of functions.

Below is a simple example that can read our DynamoDB table and it can write a new row with an updated value.

<script src=""></script>

  region: "ca-central-1",
  accessKeyId: "AKIAID7xxxxxxxxxxxx",
  secretAccessKey: "kzzeko8xxxxxxxxxxxxx"

var docClient = new AWS.DynamoDB.DocumentClient();

function createItem() {
  var d = new Date();
  var n = d.getSeconds();
    var params = {
        TableName :"my_iot_burl1",
      'signal': 'mytag4',
      'desc': 'test tag4',
      'info': {
        'quality': "questionable"
    docClient.put(params, function(err, data) {
        if (err) {
            document.getElementById('textarea').innerHTML = "Unable to add item: " + "\n" + JSON.stringify(err, undefined, 2);
        } else {
            document.getElementById('textarea').innerHTML = "PutItem succeeded: " + "\n" + JSON.stringify(params,undefined, 2);

function readItem() {
    var table = "my_iot_burl1";
    var signal = 'mytag4';
    var params = {
        TableName: table,
        Key:{'signal' : 'mytag4' }
    docClient.get(params, function(err, data) {
        if (err) {
            document.getElementById('textarea2').innerHTML = "Unable to read item: " + "\n" + JSON.stringify(err, undefined, 2);
        } else {
            document.getElementById('textarea2').innerHTML = "GetItem succeeded: " + "\n" + JSON.stringify(data, undefined, 2);


<input id="createItem" type="button" value="Create Item" onclick="createItem();" />
<textarea readonly id= "textarea" style="width:400px; height:300px"></textarea>
<input id="readItem" type="button" value="Read Item" onclick="readItem();" />
<textarea readonly id= "textarea2" style="width:400px; height:300px"></textarea>



On first impressions I was pretty overwhelmed using Amazon Web Services. However after some time playing with it I found that it wasn’t unlike many other Web services that are available.

DynamoDB can be used to communicate with sensor values from a Raspberry Pi, but unfortunately there are no mainstream libraries for Arduino projects. AWS does offer an MQTT solution that can be used to update a DynamoDB table.

For small IoT projects I feel that AWS is overkill, but for larger projects I would definitely consider using it.

RabbitMQ for IoT

For Internet of Things (IoT) projects that are a lot of different ways that the sensors, devices and client interfaces can be connected together. For many projects using simple MQTT (Message Queue Telemetry Transport) is all that you need. However if you’re trying to merge and build IoT projects that use both MQTT and AMQP (Advanced Message Queue Protocol) or require a REST API then you should take a look at RabbitMQ.

RabbitMQ is an open source middleware solution that natively uses AMQP communications but it has a good selection of plug-ins to support features like: MQTT, MQTT Web Sockets, HTTP REST API and server-to-server communications.


In this blog we will setting up a RabbitMQ server, and we will look at some of the differences between MQTT and AMQP messaging. Finally an example of an Anduino MQTT message will be presented as both an MQTT and an AMQP item in a Node-Red dashboard.

Getting Started

RabbitMQ can be installed on Window, Linux, MacOS systems and there are also some cloud based offerings. For small systems lower end hardware like a Raspberry Pi can be used.  For complete RabbitMQ installation instructions see: . To install and run RabbitMQ on a Ubuntu system enter:

sudo apt-get update
sudo apt-get install rabbitmq-server
sudo service rabbitmq-server start

The next step is to add some plug-ins. For my project I loaded the MQTT and Web Administration plug-ins:

sudo rabbitmq-plugins enable rabbitmq_mqtt
sudo rabbitmq-plugins enable rabbitmq-management

The rabbitmqctl command line tool allows you to configure and review the RabbitMQ server. To add a user admin1,  with password admin1, that has config, write and read rights for management and administrator access, enter:

sudo rabbitmqctl add_user admin1 admin1
sudo rabbitmqctl set_permissions -p / admin1 ".*" ".*" ".*"
sudo rabbitmqctl set_user_tags admin1 management administrator

After you’ve defined an administrative user the RabbitMQ web management plug-in can be accessed by: http://ip_address:15672 .


The RabbitMQ Web Management tool offers an overview of the present system load, connections, exchanges and queues. 

The RabbitMQ Web Management tool is excellent for small manual changes, if however you are looking a doing a large number of additions or changes then, rabbitmqadmin, the command line management tool can be used. This tool is  installed by:

# Get the cli and make it available to use.
sudo chmod +x rabbitmqadmin

Comparing MQTT and AMQP

It’s useful to comment about some of the differences between MQTT (Message Queue Telemetry Transport) and AMQP (Advanced Message Queueing Protocol) .

MQTT is a light weight publish-subscribe-based messaging protocol that works well with lower end hardware and limited bandwidth. For Arduino type applications where you only need to pass some sensor data MQTT is an excellent fit.

AMQP has more overhead than MQTT, because it is a more advanced protocol that includes message orientation, queuing, routing, reliability and security. Presently there are no mainstream AMQP Arduino libraries, but numerous programming options for Raspberry Pi, Linux, Windows and MacOS systems exist.  An AMQP IoT example would be to send sensor failures and alarms to dedicated maintenance and alarm queues.

MQTT and AMQP Queues

One of the biggest differences in queues is that MQTT queues are designed to show you the last available message, where as AMQP will store multiple messages in a queue.

A published MQTT message contains a message body, a retain flag and a quality of service (QoS) value.

An AMQP message can be published with added properties such as: time stamp, type of message and expiration information. AMQP messages also support the addition of custom header values. Below is a Python publish example that defines the message type to be “Pi Sensor”, and custom headers are include for status and alarm state.

#!/usr/bin/env python
import pika

node = ""
user = "pete"
pwd = "pete"

# Connect to a remote AMQP server with a username/password
credentials = pika.PlainCredentials(user, pwd)
connection = pika.BlockingConnection(pika.ConnectionParameters(node,
        5672, '/', credentials))                                    
channel =

# Create a queue if it doesn't already exist

# Define the properties and publish a message
props = pika.BasicProperties(
    headers= {'status': 'Good Quality',"alarm":"HI"},
    type ="Pi Sensor")
    routing_key='Rasp_1',body='99.5', properties = props)


For this example the Rasp_1 queue can be examined using the Queue->Get Message option in the Web Management Interface. 


RabbitMQ Messaging

There are a variety of ways that AMQP messages can be published and subscribed to. The simplest way is to create a queue and then messages can be published and subscribed to from that queue.

rabbitmq_layoutTo help with the distribution and filtering of messages AMQP supports a number of different exchange types. Messages in an exchange use bindings based on a routing key to link them to a queue. 

The main types of exchanges are: direct, fanout, headers and topic. An IoT example of a direct exchange would be if a group of Raspberry Pi sensor values are going into a “Rasp Pi Sensor” exchange. When the Rasp Pi publishes a sensor result to the exchange the message also includes a routing key to link the message to the correct queue.


An IoT example of a fanout exchange would be with critical sensor failures. The sensor failure message is sent to Bill’s and Sam’s work or task queue and the All Maintenance point queue at the same time.


Connecting MQTT

After the MQTT plug-in is installed RabbitMQ can act like a standalone MQTT broker. MQTT data can also be made available through an AMQP subscription by binding the MQTT exchange to a RabbitMQ queue.

For an MQTT project any ESP8266 supported Arduino hardware can be used. There are a number of MQTT Arduino libraries that are available. For this project I used the PubSubClient that can be installed using the Arduino Library Manager.

As a test project I used at low cost MQ-2 Combustible Gas Sensor ($3) that measures a combination of LPG, Alcohol, Propane, Hydrogen, CO and even methane. Note to fully use this sensor some calibration is required. On the MQ-2 sensor the analog signal is connected to Arduino pin A0 and the analogRead(thePin) function is used to read the sensor value.


Below is some example Arduino code required to read the MQ2 gas sensor and publish it to the RabbitMQ MQTT broker with a topic name of : mq2_mqtt.

 Basic ESP8266 MQTT publish client example
#include <ESP8266WiFi.h>
#include <PubSubClient.h>

// Update these with values suitable for your network.
const char* ssid = "your_ssid";
const char* password = "your_password";
const char* mqtt_server = ""; 
const char* mqtt_user = "admin1";
const char* mqtt_pass= "admin1";

const int mq2pin = A0; //the MQ2 analog input pin

WiFiClient espClient;
PubSubClient client(espClient);

void setup_wifi() {
  // Connecting to a WiFi network
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
  Serial.println("WiFi connected");
  Serial.println("IP address: ");

void reconnect() {
  // Loop until we're reconnected
  Serial.println("In reconnect...");
  while (!client.connected()) {
    Serial.print("Attempting MQTT connection...");
    // Attempt to connect
    if (client.connect("Arduino_Gas", mqtt_user, mqtt_pass)) {
    } else {
      Serial.print("failed, rc=");
      Serial.println(" try again in 5 seconds");

void setup() {
  client.setServer(mqtt_server, 1883);

void loop() {
  char msg[8];
  if (!client.connected()) {

  client.publish("mq2_mqtt", msg);
  Serial.print("MQ2 gas value:");


Once the MQTT value is published any MQTT client can subscribe to it.  Below is a Python MQTT subscribe example.

# - Python MQTT subscribe example 
import paho.mqtt.client as mqtt
def on_connect(client, userdata, flags, rc):
    print("Connected to broker")
def on_message(client, userdata, message):
    print ("Message received: "  + message.payload)

client = mqtt.Client()
client.username_pw_set("admin1", password='admin1')
client.connect("", 1883) 

client.on_connect = on_connect       #attach function to callback
client.on_message = on_message       #attach function to callback

client.loop_forever()                 #start the loop

Read MQTT Messages Using AMQP

MQTT clients can subscribe to MQTT messages directly or it’s possible to configure RabbitMQ to have the MQTT data accessible as AMQP. The routing of MQTT messages to AMQP queues is done using the direct exchange method.


To configure RabbitMQ to forward MQTT the following steps are done:

  1. Create a new RabbitMQ Queue – For an IoT project this would typically be a 1-to-1 mapping of the MQTT topic to a queue name.
  2. Create a binding between the MQTT Exchange and the Queue – by default all the MQTT topic go into the amq.topic exchange. For MQTT items in this exchange a binding, typically the MQTT topic name, is used as the routing key to the AMQP queue.

These steps can be done in a number of ways : manually, in the RabbitMQ config file, using the rabbitmqadmin command line tool or via a program. Because I was doing this for multiple signals I used rabbitmqadmin tool, and the syntax is:

./rabbitmqadmin declare queue name=mq2_amqp durable=true
./rabbitmqadmin declare binding source=amq.topic destination_type=queue destination=mq2_amqp routing_key=mq2_mqtt

The RabbitMQ Web Admin can be used to verify the exchange to queue binding.



The CLI tool can also be used to see if there are any values in the queue:


Node-Red Dashboard

Node-Red is a visual programming environment that allows users to create applications by dragging and dropping nodes on the screen. Logic flows are then created by connecting the different nodes together.

Node-Red has been preinstalled on Raspbian Jesse since the November 2015. Node-Red can also be installed on Windows, Linux and OSX.  To install and run Node-Red on your specific system see

To install the AMQP components, select the Manage palette option from the right side of the menu bar. Then search for “AMQP” and install node-red-contrib-amqp. If your installation of Node-Red does not have dashboards installed, search for: node-red-dashboard.


For our Node-Red MQTT and AMQP example we will use a mqtt and a amqp node from the input palette group, and two gauge nodes from the dashboard group.  The complete Node-Red logic for this is done in only 4 nodes!


Nodes are added by dragging and dropped them into the center Flow sheet. Logic is created by making connection wires between inputs and output of a node. After the logic is laid out, double click on each of the nodes to configure their specific properties. You will need to specify the MQTT and AMQP definitions of your RabbitMQ IP address,user rights, MQTT topic and AMQP queue name. You will also need to double click on the gauge nodes to configure the look-and-feel of the web dashboard.

After the logic is complete, hit the Deploy button on the right side of the menu bar to run the logic.  The Node-Red dashboard user interface is accessed by: http://ipaddress:1880/ui. 

For my project I used a number of different MQ sensors and inputs. Below is a picture of Node-Red web dashboard that we created with the same MQTT value being shown natively and as a AMQP queued value.


Final Comments

I found that RabbitMQ was easy to install and the Web Administration plug-in along with the rabbitmqadmin command line tool made it very easy to maintain. 

If you’re just looking to show sensor values then a basic MQTT broker might be all you need. However if you’re looking at some future applications like alarm, maintenance or task lists then AMQP exchanges and queue make RabbitMQ an interesting option.

More on RabbitMQ:

RabbitMQ REST API – remote interfacing (Javascript and Python examples)

RabbitMQ connections with Excel