PySimpleGUI – quick and easy interfaces

The Python PySimpleGUI project has two main goals, a simpler method for creating graphical user interfaces (GUIs), and common code for Tkinter, QT, xW and Web graphics.

I feel comfortable doing my own Python Tkinter and Web interfaces, but using common code for both local interfaces and Web apps could be extremely useful for Rasp Pi projects.

In the this blog I wanted to introduce PySimpleGUI by creating a local GUI/Web interface to control a Raspberry Pi Rover, all in less than 60 lines of code.

Getting Started with PySimpleGUI

The Python PySimpleGUI project has a number of “ports” or versions. The main version is for Tkinter based graphics and it is very fully featured. The versions for Qt,Wx and Web graphics are still in development so some testing may be required if you are hoping for full code compatibility between the different libraries.

There probably aren’t a lot of cases where you would want to flip between Qt, Wx and Tkinter graphic engines but it is remarkable that the possibility exists.

To install the default Tktinter  version of  Pysimplegui enter:

pip install pysimplegui

PySimplegui has a wide range of graphic widgets or elements. Graphic presentations are built by creating a layout variable. Graphic elements are placed in separate rows by open and closed square brackets.

gui_layout

A Button Interface Project

For my rover project I used a layout of 5 rows. The first row contains a feedback text, then rows 2-5 contains buttons.

The code below is a simple button app.


# Create a simple graphic interface
#
import PySimpleGUI as sg

layout = [ [sg.Text("the feedback" , key="feedback")],
           [sg.Button("FORWARD")],
           [sg.Button("LEFT"),sg.Button("RIGHT")],
           [sg.Button("STOP")],
           [sg.Button("QUIT")]
          ]
# Create the Window
window = sg.Window('My First App', layout)
# Event Loop to process "events"
while True:
    event, values = window.read()
    window['feedback'].Update(event) # show the button in the feedback text
    print(event,values)
    if event in (None, 'QUIT'):
        break
window.close()

The PySimplegui sg.window() call displays a window with the title and a layout definition (line 11). The window.read() will return events and values that have been changed (line 14). The feedback text element (line 5) is given a key name of  feedback, and this key name is used for updates to show the key press (line 15).

sg_basic

Standalone Web Apps with PySimpleGUIWeb

The PySimpleGUIWeb library is still under development, so be aware that not all the features in PySimpleGUI are fully supported in the Web version. PySimpleGUIWeb is an excellent way to create a lightweight standalone Web interface, but it is important to note that it isn’t designed to be a multi-page/multi-user Web environment.

To install PySimpleGUIWeb enter:

pip install remi
pip install pysimpleguiweb

The PySimpleGUIWeb window() call has a few more options, such as:

  • web_ip – the IP address to use for the PySimpleGUIWeb micro Web server
  • web_port – port on the micro Web server
  • web_start_browser – open a Web browser on app start

If you use our earlier button example but this time import PySimpleGUIWeb and add some web options we see an almost identical presentation however this time it’s in a Web interface.

sg_basic_web

Command line options can be used to toggle between the different libraries by:

import sys

# Pass any command line argument for Web use 
if len(sys.argv) > 1: # if there is use the Web Interface 
    import PySimpleGUIWeb as sg
    mode = "web"
else: # default uses the tkinter GUI
    import PySimpleGUI as sg
    mode = "tkinter"

Formatting of Display Elements

The next step is to adjust the graphic elements’  fonts, colors, and size properties.

Below is an example of changing the “FORWARD” button to have a size of 32 characters wide and 3 lines high with color and larger font.

[sg.Button("FORWARD", size=(32,3), 
  font="Ariel 32", 
  button_color=('white','green'))]

To make the interface more usable all the rover control buttons can be adjusted and the “QUIT” button can be left the default.

py_rovergui

Raspberry Pi Rover Interface

For my Raspberry Pi Rover project I used :

  • Arduino car chassis (~ $15),
  • a portable USB charger
  • Pimoroni Explorer Hat Pro (a Pi motor shield)

Below is the final code and it used a command line option (any character) to toggle into a Web application, otherwise it was the default PySimpleGUI interface. The application also included the Pi GPIO library to start/stop the car chassis motors.


# SGui_rover.py - use PySimpleGUI/Web to control a Pi Rover Pi
#

import sys
# Pass any command line argument for Web use
if len(sys.argv) > 1: # if there is use the Web Interface
    import PySimpleGUIWeb as sg
    mode = "web"
else: # default uses the tkinter GUI
    import PySimpleGUI as sg

import RPi.GPIO as gpio
gpio.setmode(gpio.BOARD)
# Define the motor pins to match your setup
motor1pin = 38 # left motor
motor2pin = 37 # right motor
gpio.setup(motor1pin, gpio.OUT)
gpio.setup(motor2pin, gpio.OUT)

# Send Action to Control Rover
def rover(action):
if action == "FORWARD":
    gpio.output(motor1pin, gpio.HIGH)
    gpio.output(motor2pin, gpio.HIGH)
if action == "LEFT":
    gpio.output(motor1pin, gpio.HIGH)
    gpio.output(motor2pin, gpio.LOW)
if action == "RIGHT":
    gpio.output(motor1pin, gpio.LOW)
    gpio.output(motor2pin, gpio.HIGH)
if action == "STOP":
    gpio.output(motor1pin, gpio.LOW)
    gpio.output(motor2pin, gpio.LOW)

# All the stuff inside your window.
myfont = "Ariel 32"
layout = [ [sg.Text(" ",size=(20,1) , key="feedback")],
[sg.Button("FORWARD", size=(32,3), font=myfont, button_color=('white','green'))],
[sg.Button("LEFT", size=(15,3), font=myfont),sg.Button("RIGHT", size=(15,3), font=myfont)],
[sg.Button("STOP", size=(32,3), font=myfont, button_color=('white','red'))],
[sg.Button("QUIT")]
]
# Create the Window
if mode == "web":
    window = sg.Window('PySimpleGUI Rover Control', layout,
        web_ip='192.168.0.106', web_port = 8888, web_start_browser=False)
else:
    window = sg.Window('PySimpleGUI Rover Control', layout )

# Event Loop to process "events" and pass them to the rover function
while True:
    event, values = window.read()
    print(event,values)
    if event in (None, 'QUIT'): # if user closes window or clicks cancel
        break
    window['feedback'].Update(event) # show the button in the feedback text
    rover(event)

window.close() # exit cleanly

py_sg_rover

Final Comment

I feel that PySimpleGUI and PySimpleGUIWeb have a lot of great potential for Raspberry Pi projects.

 

littleBit Dashboards (without Cloud Bits)

littleBits is a set of electronic components that magnetically connect together. litteBits is geared towards the kids STEP market and it is available in many schools and libraries.

The littleBits company has done an excellent job making their product easy to use. There is a large variety of different “bit” modules and for Internet applications there is a Cloud Bit ($59).

I found that the Cloud Bit was very easy to get up and running, but I found it was expensive at $59 and somewhat limiting, for example you are only access 1-input and 1-output. So if you want to do 2 inputs/output you would need to purchase a second Cloud bit module.

In this blog I’d like to document how I used a $39 Arduino Bit to do 3-inputs and 3-outputs. I also had the code talk directly to a free Web Dashboard (AdaFruit).

littleBits Arduino Program

A set of commands needs to be setup between the littlebits Arduino module and the PC/Pi. In my Arduino program I referenced the ports A,B,C as inputs (on the left side), and D,E,F as outputs (on the right side).

The commands from the PC/Pi would be : reference_pin:value, for example D:255 would set the D (top left pin) at 100%. It’s important to note that Arduino inputs and outputs are scaled from 0-255.

For inputs the littleBits would send the results as pin: reference_pin:value, for example B:255 would be the result at full scale for the A0 input.

ard_abc

My  test setup had:

  • A fork bit – this meant I only needed 1 power input source
  • A slider bit (0-1) on bit D0 (A)
  • A dimmer bit (0-255) on bit A0 (B)
  • A temperature bit on bit A1 (C)
  • An LED on bit d1 (D)
  • A number bit on D5 (E)
  • a bargraph bit on D9 (F)

lb_ard_setup

Below is the littleBits Arduino program that managed the serial communications.

// littleBits_2_Dashboards - create a serial interface to read/write to a PC/Pi
//
// Command from the littleBits: (A,B,C are the left pins) 
//  A:value <- for example B:24, pin A0 (2nd input) is 24 

// Commands from the PC/Pi: (D,E,F are the right pins)
//  D:output <- for example E:128, set pin A0 to 50% (128/255)
//
String thecmd; 
String thevalue;
String theinput;
char outstring[3];

void setup() {
  //define the littleBits right side pins 1,5 and 9 
  pinMode(1, OUTPUT);
  pinMode(5, OUTPUT);
  pinMode(9, OUTPUT);
  // define the littleBits left side inputs
  pinMode(0, INPUT);
  pinMode(A0, INPUT);
  pinMode(A1, INPUT);
  
  Serial.begin(9600); // this needs to match the PC/Pi baud rate
}
void loop() { 
  if (Serial.available() > 0) {
    thecmd = Serial.readStringUntil("\n"); 
    if (thecmd.length() > 2) { // ensure the msg size is big enough
      thevalue = thecmd.substring(2);
      if (thecmd.startsWith("D")) { analogWrite(1,thevalue.toInt()); }
      if (thecmd.startsWith("E")) { analogWrite(5,thevalue.toInt()); }
      if (thecmd.startsWith("F")) { analogWrite(9,thevalue.toInt()); }
    }     
  }
  // Try 3 different inputs: d0 = on/off , A0 = pot, A1 = temp sensor

  sprintf(outstring,"%d",digitalRead(0));
  Serial.write("A:");
  Serial.write(outstring);
  Serial.write("\n");

  sprintf(outstring,"%d",analogRead(A0));
  Serial.write("B:");
  Serial.write(outstring);
  Serial.write("\n");

// A1 is an "i12" littleBits temperature sensor
  int temp = analogRead(A1);
  temp = map(temp,0,1023,0,99); //rescale. Sensor range is 0-99 C or F
  sprintf(outstring,"%d",temp);
  Serial.write("C:");
  Serial.write(outstring);
  Serial.write("\n");
  

  delay(5000);
}

The Arduino IDE “Serial Monitor” can be used to view the output and set values.

msgbox

Python on the PC or Raspberry Pi

The Arduino program will send input data for A,B,C every 5 seconds. This input can be seen in Python by:

#
# littleBits Read Test
#
import serial

ser = serial.Serial(port='/dev/ttyACM1', baudrate=9600) # format for Linux
#ser = serial.Serial(port='COM1', baudrate=9600) # format for Windows

while True:
    inline = ser.readline()
    inline = inline.decode() # make a string
    pin = inline[0:1] # the first character is the pin
    thevalue = inline[2:-1] # the value is between ":" and "\n"<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>
    print(pin,thevalue)

The output will look something like:

A  1
B  1023
C  21

To write commands from Python:

Write
#
# littleBits Write Test
#
import serial

ser = serial.Serial(port='/dev/ttyACM2', baudrate=9600) # format for Linux
#ser = serial.Serial(port='COM1', baudrate=9600) # format for Windows

while True:
    print("\nWrite an output value to littleBit")
    out = input("Enter pin:value, pin=A,B,C example: 'E:255' : ")
    out = out.upper() + "\n"
    out2 = out.encode('utf_8')
    ser.write(out2)

Adafruit Dashboards

There are lots of good free dashboards. For this project I used the Adafruit site. To get started you will need to log in and create a free account.

I’ve bought a number of components from Adafruit. I think that they are an excellent company that goes out of their way to create great user guides and products.

To get started with Adafruit Dashboards see: https://github.com/adafruit/Adafruit_IO_Python

The first step is to add some Adafruit tags that the code can read/write to.

Ada_feeds

In the Python code a couple of dictionaries (lb_inputs, lb_outputs)  were created to link the littleBit references (A-F) with the Adafruit tags. Also two dictionaries (lb_inlast, lb_outlast) are used to minimize communications traffic so that only new values were written.

#
# Import standard python modules
import time, random
import serial

# import Adafruit IO REST client
from Adafruit_IO import Client, Feed, RequestError

ADAFRUIT_IO_USERNAME = "put_your_username_here"
ADAFRUIT_IO_KEY = "c039f24ecb6...xxxxx"

aio = Client(ADAFRUIT_IO_USERNAME, ADAFRUIT_IO_KEY)

# Create dictionaries of inputs, output, and last values
lb_inputs = {"A":"lb-slide", "B":"lb-dimmer","C": "lb-temp"}
lb_inlast = {"A":0, "B":0,"C": 0}
lb_outputs = {"D":"lb-led", "E":"lb-number", "F":"lb-bar"}
lb_outlast = {"D":0, "E":0,"F": 0}

# Setup the serial port connection
ser = serial.Serial(port='/dev/ttyACM1', baudrate=9600)

while True:
    # Get values from littleBits and write to the dashboard
    inline = ser.readline()
    inline = inline.decode() #inline should look like: A:125\n
    pin = inline[0:1] # pin is the first character in string
    thevalue = inline[2:-1] # value is between ":" and "\n"
    if lb_inlast[pin] != thevalue: # Write only new values
        print(pin,thevalue, lb_inputs[pin])
        ada_item = aio.feeds(lb_inputs[pin])
        aio.send(ada_item.key,thevalue)
        lb_inlast[pin] = thevalue

    thetag = 'lb-slide'
    # Write new dash values to littleBits if they've changed
    for lbtag, dashtag in lb_outputs.items():
        print (lbtag,dashtag)
        thevalue = aio.receive(dashtag).value
        if lb_outlast[lbtag] != thevalue: # Write only new values
            outstr = lbtag + ":" + thevalue + "\n"
            print(outstr)
            ser.write(outstr.encode('utf_8'))
            lb_outlast[lbtag] = thevalue   

    time.sleep(2)

If everything is working correctly then new values should be written to in both directions. On the Adafruit Web site the Feed page should show the new values.

To make things look more presentable Adafruit Dashboards can be used.

ada_dash

Final Comments

In this project I used the Adafruit API, other good platforms would be IFTTT and Node-Red

Arduino Yún for IoT

There are some great Internet of Things hardware platforms, such as Raspberry Pi’s, ESP-32 based Arduino modules, Particle and Beaglebone to name just a few. Each of these systems have their strengths and weakness. Some are strong on the hardware side, like the Arduino modules, and others excel on the programming side like the Raspberry Pi.  

The Yún is somewhat unique in that it’s a module with two processors, one that supports standard Arduino programming and a second processor that supports Linux and the OpenWrt wireless stack. The Yún ($59) has an Arduino Uno form factor and there are clones like the LinkIt Smart 7688 Duo ($18) in an Arduino Nano form factor.

yun_overview

In this blog I wanted to document some of key feature and functions that I worked through, namely:

  • moving file – scp and ftp
  • Python bridging to Arduino
  • uhttpd Web Server – with Python CGI
  • Yún REST API
  • MQTT
  • Yún mailbox

Yún Overview

The Yún, now in revision 2, seems to have been somewhat overlooked because of the all the low cost ESP-8266 and ESP32 based Arduino modules starting around $2.

Some the interesting features of the Yún include:

  • Arduino code can launch and get feedback from Linux apps
  • A bridge library allows Linux programs and Arduino to share data
  • Arduino libraries for:
    • Web client
    • Web server
    • Mailbox to Linux
  • Yún has a lightweight web server, uhttpd, that can be used to Python, PHP, Lua CGI web programs
  • Yún has a read/write REST API web service

Getting Started

The Arduino side of the Yún is available “out of the box” like any other Arduino module when you connect a USB cable into the module.

The Linux side however requires some configuration. Please see one of the guides on the Linux setup. Once your module is connected to your network, the Yun webserver can be used to configure features and add software components.

yun_software1

Another option for loading software is to use the opkg package manager from an SSH connection.

yun_opkg

So for example to install nano, the simple text editor, enter:

 opkg install nano

Moving File and Working in OpenWrt Linux

The OpenWrt Linux isn’t loaded with with an X-windows environment and this means that you can not run idle or leafpad etc. to do easy editing of program files. Nano is a good command line text editor but it’s not the same as a windows based editor.

To move files between a PC and OpenWrt you have some options. The two that I like are:

  • scp – secure copy, this is built in both OpenWrt and Microsoft Window 10
  • ftp – file transfer protocol. Needs to be installed in OpenWrt.

There are a few ftp servers that could be installed in OpenWrt, vsftp is a lightweight option and it can be installed by:

opkg update
opkg install vsftpd

Once vsftpd is installed it needs to be enabled, this can be done from the command line or via the web interface.

yun_ftp_start.png

Yún Bridge

The Yun Arduino bridge library allow variables to be passed between the Arduino code and a Python program. Below is an example that writes two random variables available, and it creates a “bridge1” variable that can be written from the Python side.

// Simple Yun Bridge Example
//

#include <Bridge.h>
// create a bridge variable to get remote data
char bridge_Value[10];

void setup()
{
    Bridge.begin();     // this launches /usr/bin/run-bride on Linino
}

void loop()
{    
    // create 2 random bridge values, that are sourced from the Arduino
    Bridge.put("random1", String(random(1, 100)));
    Bridge.put("random2", String(random(1, 100)));
    // Called the bridge value "bridge1". This is the name used on the Python side
    Bridge.get("bridge1", bridge_Value, 6);
    delay(1000);  
}

An example to read/write in Python:


#!/usr/bin/python

import sys

sys.path.insert(0, '/usr/lib/python2.7/bridge/')

from bridgeclient import BridgeClient as bridgeclient
value = bridgeclient()

message = value.get("random1") # get a value from Arduino

print "Random1: ", message

value.put("bridge1","1111")  # set a value to Arduino

Yún uhttpd web server

The uhttp web server is used for Yun setup and configuration. This web server can also be used to for custom static pages and user web apps. To view/modify the web server settings:

nano /etc/config/uhttpd

Within this config file, you can enable custom web applications by defining an interpreter:

# List of extension->interpreter mappings.
# Files with an associated interpreter can
# be called outside of the CGI prefix and do
# not need to be executable.
# list interpreter ".php=/usr/bin/php-cgi"
# list interpreter ".cgi=/usr/bin/perl"
list interpreter ".py=/usr/bin/python

# Lua url prefix and handler script.
# Lua support is disabled if no prefix given.
# option lua_prefix /luci
# option lua_handler /usr/lib/lua/luci/sgi/uhttpd.lua

The default directory for user programs is: /www/cgi-bin

Python CGI – Get Values

To read the Arduino bridge values in a Python CGI program, add a file to the /www/cgi-bin directory. For my example I called the file p1.py :


#!/usr/bin/python

import sys
import cgi

sys.path.insert(0, '/usr/lib/python2.7/bridge/')

from bridgeclient import BridgeClient as bridgeclient
value = bridgeclient()

print "Content-type:text/html\r\n\r\n"
print '<html>'
print '<head>'
print '<title>Python CGI Bridge</title>'
print '</head>'
print '<body>'
print '<h2>Python CGI Bridge</h2>'
print 'Random Value1: ' + value.get("random1")
print 'Random Value2: ' + value.get("random2")
print '</body>'
print '</html>'

To run the python script on the web page you will need to change the file right to executable:

chmod +x p1.py

You can debug and see output from the command line.


root@yun1:/www/cgi-bin# ./p1.py
Content-type:text/html

<html>
<head>
<title>Python CGI Bridge</title>
</head>
<body>
<h2>Python CGI Bridge</h2>
Random Value1: 13
Random Value2: 24
</body>
</html>

If the output looks good, try the app from the Web page:

CGI_bridge1

Python CGI – Put Values

There are a number of methods that can be used to send user input from a web page. A simple approach is to use a form. The form data can be read from the cgi.FieldStorage object, using a form.getvalue() call.

#!/usr/bin/python

# Import modules for CGI handling
import cgi
import sys

sys.path.insert(0, '/usr/lib/python2.7/bridge/')
from bridgeclient import BridgeClient as bridgeclient
value = bridgeclient()

# Create instance of FieldStorage
form = cgi.FieldStorage()

# Get data from fields
bridge1 = form.getvalue('bridge1')
if bridge1 != "" :
    value.put("bridge1",bridge1)


print "Content-type:text/html\r\n\r\n"
print """
<html>
<head>
<title>Python CGI - Put Bridge Value</title>
</head>
<body>
<h2>Python CGI Form - Put Bridge Value</h2>
<form action = "/cgi-bin/p2.py" method = "post">
Enter BRIDGE1 value: <input type = "text" name = "bridge1"><br />

<input type = "submit" value = "Submit" />
</form>
</body>
</html>"""

 

The web page will call itself when the submit button is pressed.

CGI_bridge2

Yún REST API

The Yún REST API is a web service that allow remote users and web browsers to view and set bridge values. I found the REST API to be a good tool for testing my Web CGI and Python applications

To view all bridge variables enter: http://yourYun_IP/data/get

yun_rest_1

To get a specific bridge value enter:  http://yourYun_IP/data/get/my_bridge_value

yun_rest_2

To put a bridge value enter: http://yourYun_IP/data/put/my_bridge_value/value 

yun_rest_3

IoT Connections – MQTT

For Internet of Things (IoT) projects you need to pass data from the Arduino and some server. The communications and the server would be something like MQTT or Redis.

The Yún does not have direct access to the Wifi or ethernet port so the standard Arduino libraries for MQTT or Redis etc. will not work. An alternative approach is load the protocol’s command line client on the Linux side and then Arduino can shell out to the Linux tool.

For example to load the  Mosquitto MQTT command line tools:

opkg update
opkg install mosquitto-client

To test MQTT publishing to a topic (mytag1) with a message of 249 on remote client (192.168.0.116) :

mosquitto_pub -h 192.168.0.116 -u username -P password -t mytag1 -m 249

To remotely subscribe to

mosquitto_sub -h 192.168.0.116 -u username -P password -t mytag1

An example of sending out four MQTT messages in Arduino:

/*
Shell out to pass values to a MQTT command line pub
*/
#include <Process.h>

void setup() {
  Bridge.begin();	// Initialize the Bridge
  Serial.begin(9600);	// Initialize the Serial

  // Wait until a Serial Monitor is connected.
  while (!SerialUSB);
}

void loop() {
  Process p;
  String thecmd;
  String strval = String( random(0,100));
  // create a string with host, username and password, -t is for the topic
  String theparams = "-h 192.168.0.116 -u pete -P pete -t ";
  int numtopics = 4;
  String topics[4] = {"tag1","tag2","tag3","tag4"};

  for (int i=0; i < numtopics ; i++) {
    strval = String( (i*100) + random(1,99)); // create a random value - 0,100+,200+,300+
    thecmd = "mosquitto_pub " + theparams + topics[i] + " -m " + strval;
    Serial.println(thecmd);
    p.runShellCommand(thecmd);
    // do nothing until the process finishes, so you get the whole output:
    while (p.running());
  }
  // Wait 5 seconds and repeat
  delay(5000);  // wait 5 seconds before you do it again
}

It is also possible to create some bridge variables and pass them to a Python program that could do the MQTT communications.

Yún Mailbox

At present the mailbox only works in one direction, and this into the Arduino. Both the REST web interface and Python have writing functionality but no read capabilities (on the forums this has been identified, so a future revision may add this).

To write using REST:

http://yun_ip/mailbox/mymessagetext

To write using Python:


import sys
sys.path.insert(0, '/usr/lib/python2.7/bridge')
from bridgeclient import BridgeClient
client = BridgeClient()
client.mailbox("my_message")

The Arduino code to read message:

// Mailbox Read Example
//
#include <Mailbox.h>

void setup() {

  // Initialize Bridge and Mailbox
  Bridge.begin();
  Mailbox.begin();

  // Initialize Serial
  SerialUSB.begin(9600);

  // Wait until a Serial Monitor is connected.
  while (!SerialUSB);
  SerialUSB.println("Mailbox Read Message\n");
}

void loop() {
  String message;

  // if there is a message in the Mailbox
  if (Mailbox.messageAvailable()) {
    // read all the messages present in the queue
    while (Mailbox.messageAvailable()) {
      Mailbox.readMessage(message);
      SerialUSB.println(message);
    }

    SerialUSB.println("Waiting 10 seconds before checking the Mailbox again");
  }
  // wait 30 seconds
  delay(10000);
}

I’m not totally sure when I’d use a mailbox. The mailbox is a generic message and it’s not queued (only 1 message) so I think that using the standard bridging of values with get/put is more useful.

Final Comments

The Yun is missing a lot of networking libraries that are available with the ESP-8266 and ESP32 family of Arduino modules. In some cases like with MQTT that was a good Linux command line tool that could be used, but in cases like where you want to connect to a Redis server, you might have some challenges.

I really liked Yun’s built in uhttpd web server, this is far superior to the ESP-8266 Arduino web server library calls.

I found that putting logic into a combination of Python and Arduino could be a little confusing, but for projects with a lot of text or json data using Python would be a real plus. Also for projects where multitasking is required use the Linux process would be ideal.

RabbitMQ REST API

RabbitMQ is an open source middleware message solution that natively uses AMQP communications but it has a good selection of plug-ins to support features like: MQTT, MQTT Web Sockets, HTTP REST API and server-to-server communications.

rabbitmq_overview

RabbitMQ has a very good selection of interfacing clients for most of the major programming languages. For applications where there is no custom programming library the REST API can be used.

Applications where the REST API could be used might include:

  • Arduino IoT projects
  • Web Clients (not using NodeJS)
    • For simple reading and writing of messages
    • For monitoring of RabbitMQ loading
  • AppInventor Android apps

Unfortunately I didn’t find the RabbitMQ REST API to be 100% straightforward, so in this blog I could like to document some of my notes and testing.

What is REST API

Representational State Transfer (REST) is a software style for creating Web services. A client issues an HTTP request to a server with a GET/POST/PUT or DELETE method and the server responds with a JSON (Javascript Object Notation) formatted message.

RESTAPI_overview

The most common tool to test REST API is curl. Curl is supported on all major OS’s. The curl syntax will vary slightly between Windows and Linux, especially in how the systems handle strings.

RabbitMQ Setup

RabbitMQ can be installed on Window, Linux, MacOS systems and there are also some cloud based offerings. For small systems lower end hardware like a Raspberry Pi can be used.  For complete RabbitMQ installation instructions see: https://www.rabbitmq.com/download.html . To install and run RabbitMQ on a Ubuntu system enter:

sudo apt-get update
sudo apt-get install rabbitmq-server
sudo service rabbitmq-server start

The next step is to add some plug-ins. For the Web Administration and REST API plug-ins:

sudo rabbitmq-plugins enable rabbitmq-management

The rabbitmqctl command line tool allows you to configure and review the RabbitMQ server. To add a user pete,  with password pete, that has config, write and read rights for management and administrator access, enter:

sudo rabbitmqctl add_user pete pete
sudo rabbitmqctl set_permissions -p / pete ".*" ".*" ".*"
sudo rabbitmqctl set_user_tags pete management administrator

After you’ve defined an administrative user the RabbitMQ web management plug-in can be accessed by: http://ip_address:15672 .

Rabbit_webadmin

Once the Web Management component is installed it is much easier to test and play with REST API calls.

REST API Calls Using CURL

For a complete set of RabbitMQ REST API calls see: https://pulse.mozilla.org/api/ .

Typically you would use the RabbitMQ tools to create and exchanges and queues but it is possible to do this via the REST API interface.

Below are some examples using user: pete and a password of pete . Note: %2f is used for default (blank name) virtual host.

Create an exchange: webex

curl -i -u pete:pete -H “content-type:application/json” -X PUT http://192.168.0.116:15672/api/exchanges/%2f/webex -d”{“”type””:””direct””,””auto_delete””:false,””durable””:true,””internal””:false,””arguments””:{}}”

Create a Queue: webq1

curl -i -u pete:pete -H “content-type:application/json” -X PUT http://192.168.0.116:15672/api/queues/%2f/webq1 -d”{“”auto_delete””:false,””durable””:true,””arguments””:{}}”

Create Binding: Bind queue webq1 to exchange webex with a routing key of webq1

curl -i -u pete:pete -H “content-type:application/json” -X POST http://192.168.0.116:15672/api/bindings/%2f/e/webex/q/webq1 -d”{“”routing_key””:””webq1“”,””arguments””:{}}”

Comments: you need to bind a queue to an exchange if you wish to write to it. The routing key does the binding. The key can be any name but using the queue name makes it simple. A queue can also have binding to multiple exchanges.

Get a message: from queue webq1

curl -i -u pete:pete -H “content-type:application/json” -X POST http://192.168.0.116:15672/api/queues/%2f/webq1/get -d”{“”count””:5,””requeue””:true,””encoding””:””auto””,””truncate””:50000}”

Comments: A queue can have multiple messages. A requeue=false, will remove messages that are already read.

Post a message (44.4) : to exchange webex with routing key webq1

curl -i -u pete:pete -H “content-type:application/json” -X POST http://192.168.0.116:15672/api/exchanges/%2f/webex/publish -d”{“”properties””:{},””routing_key””:””webq1“”,””payload””:””44.4“”,””payload_encoding””:””string””}”

 

Javascript REST API Client

I had some Web browser issues when my test PC and my RabbitMQ server were in different domains. Web Browsers like Chrome and  Firefox will complain about a CORS policy problem.

Cross-origin resource sharing (CORS) is a mechanism that allows restricted resources on a web page to be requested from another domain outside the domain from which the first resource was served.

If you are crossing domains and you want to do some quick testing Microsoft Explorer will work.

Javascript GET a Queue Message

I used the AJAX XMLHttpRequest object to do my GET and POST requests.

<!DOCTYPE html>
<html>
<body>
<h2>RabbitMQ REST API - Get Queue Message</h2>

URL/API Call: <input type="text" id="theurl" value="http://192.168.0.116:15672/api/queues/%2f/"  size = "50" /> <br><br>
Queue Name:   <input type="text" id="qname" value="webex"/> <br><br>
Requeue: <input type="text" id="rq" value="true" /> (false = remove old values)<br><br>
Message Count: <input type="text" id="mcount" value="5"/> <br><br>
<button type="button" onclick="sendoutput()">Get Message(s)</button>
<br>

<hr>
<p id="demo"></p>

<script>
function sendoutput() {
	// RabbitMQ username and password
	var username = "pete";
	var password = "pete";
	var xhttp = new XMLHttpRequest();
	
    var fullurl = document.getElementById("theurl").value + document.getElementById("qname").value + "//get";
	
	xhttp.open("POST", fullurl, true,username,password);
	
	// change to your Auth username and password

	xhttp.setRequestHeader("Content-Type","application/jsonp");

	var rq = document.getElementById("rq").value;
	var mcount = document.getElementById("mcount").value;
	var params = JSON.stringify({"count":mcount,"requeue":rq,"encoding":"auto","truncate":"50000"});
	// Define a handler function when the response comes back
	xhttp.onreadystatechange = function() {
		// Get the response and put on it the screen
		if (this.readyState == 4 ) {	
			document.getElementById("demo").innerHTML = "Result=" +xhttp.responseText;
		}
	}

	xhttp.send(params);
}

</script>
</body>
</html>

r_get

Javascript POST an Exchange Message

Unfortunately you can not send a message directly to a queue, instead you need to send a message to an exchange with routing key. This routing key is linked to the queue that you wish to connect to.

<!DOCTYPE html>
<html>
<body>
<h2>RabbitMQ REST API - An Exchange Message</h2>

URL/API Call: <input type="text" id="theurl" value="http://192.168.0.116:15672/api/exchanges/%2f/"  size = "50" /> <br><br>
Exchange Name:   <input type="text" id="exname" value="webex"/> <br><br>
Routing Key:   <input type="text" id="rkey" value="webq1"/> <br><br>
Value to Send:   <input type="text" id="theval" value="11.1"/> <br><br>

<button type="button" onclick="sendoutput()">Send Output</button>
<br>

<hr>
<p id="demo"></p>

<script>
function sendoutput() {
	// RabbitMQ username and password
	var username = "pete"; 
	var password = "pete";  
	
	var xhttp = new XMLHttpRequest();
	
	var fullurl = document.getElementById("theurl").value + document.getElementById("exname").value + "//publish";
	
	xhttp.open("POST", fullurl, true,username,password);

	xhttp.setRequestHeader("Content-Type","application/jsonp");
	
	var rkey = document.getElementById("r").value
	var theval = document.getElementById("theval").value
	var params = JSON.stringify({"properties":{},"routing_key":rkey,"payload":theval,"payload_encoding":"string"});
	
	// Define a handler function when the response comes back
	xhttp.onreadystatechange = function() {
		// Get the response and put on it the screen
		if (this.readyState == 4 ) {	
			document.getElementById("demo").innerHTML = "Result=" +xhttp.responseText;
		}
	}
	xhttp.send(params);
}
</script>
</body>
</html>

r_post

Python – get message(s) from a queue

The requests library is extremely simple to use for REST interfacing. It is installed by:

pip install requests

# importing the requests library
import requests

# defining the api-endpoint
API_ENDPOINT = "http://192.168.0.116:15672/api/queues/%2f/webq1/get"

headers = {'content-type': 'application/json'}
# data to be sent to api
pdata = {'count':'5','requeue':'true','encoding':'auto','truncate':'50000'}

# sending post request and saving response as response object
r = requests.post(url = API_ENDPOINT ,auth=('pete', 'pete'), json = pdata, headers=headers)

# extracting response text
pastebin_url = r.text
print("Response :%s"%pastebin_url)

Python – Write a message to an exchange


# importing the requests library
import requests
import json

# defining the api-endpoint
API_ENDPOINT = "http://192.168.0.116:15672/api/exchanges/%2f/webex/publish"

# your source code here

headers = {'content-type': 'application/json'}
# data to be sent to api

pdata = {'properties':{},'routing_key':'webq1','payload':'90.4','payload_encoding':'string'}

# sending post request and saving response as response object
r = requests.post(url = API_ENDPOINT ,auth=('pete', 'pete'), json = pdata, headers=headers)

# extracting response text
pastebin_url = r.text
print("Response :%s"%pastebin_url)

Final Comments

The RabbitMQ REST interface is not designed for high throughput, but it is an excellent way to connect devices that have no native API support.

For my next round of testing I’d like to work on an Arduino and a AppInventor Android app.

Apache Kafka with Node-Red

Apache Kafka is a distributed streaming and messaging system. There are a number of other excellent messaging systems such as RabbitMQ and MQTT. Where Kafka is being recognized is in the areas of high volume performance, clustering and reliability.

Like RabbitMQ and MQTT, Kafka messaging are defined as topics. Topics can be produced (published) and consumed (subscribed). Where Kafka differs is in the storage of messages. Kafka stores all produced topic messages up until a defined time out.

Node-Red is an open source visual programming tool that connects to Raspberry Pi hardware and it has web dashboards that can be used for Internet of Things presentations.

In this blog I would like to look at using Node-Red with Kafka for Internet of Things type of applications.

Getting Started

Kafka can be loaded on a variety of Unix platforms and Windows.  A Java installation is required for Kafka to run, and it can be installed on an Ubuntu system by:

apt-get install default-jdk

For Kafka downloads and installation instructions see: https://kafka.apache.org/quickstart. Once the software is installed and running there a number of command line utilities in the Kafka bin directory that allow you to do some testing.

To test writing messages to a topic called iot_test1, use the kafka-console-producer.sh  command and enter some data (use Control-C to exit):

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic iot_test1
11
22
33

To read back and listen for messages:

 bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic iot_test1 --from-beginning
11
22
33

The Kafka server is configured in the /config/server.properties  file. A couple of the things that I tweeked in this file were:

# advertised the Kafka server node ip
advertised.listeners=PLAINTEXT://192.168.0.116:9092
# allow topics to be deleted
delete.topic.enable=true

Node-Red

Node-Red is a web browser based visual programming tool, that allows users to create logic by “wiring” node blocks together.  Node-Red has a rich set of add-on components that includes things such as: Raspberry Pi hardware, Web Dash boards, email, Tweeter, SMS etc.

Node-Red has been pre-installed on Raspbian since 2015. For full installation instructions see:  https://nodered.org/#get-started

To add a Node-Red component select the “Palette Manager”, and in the Install tab search for kafka. I found that the node-red-contrib-kafka-manager component to be reliable (but there are others to try).

For my test example I wanted to create a dashboard input that could be adjusted. Then read back the data from the Kafka server and show the result in a gauge.

This logic uses:

  • Kafka Consumer Group – to read a topic(s) from a Kafka server
  • Dashboard Gauge – to show the value
  • Dashboard Slider – allows a user to select a numeric number
  • Kafka Producer – sends a topic message to the Kafka server

nodered_kafka Double-click on the Kafka nodes and in the ‘edit configuration’ dialog create and define a Kafka broker (or server). Also add the topic that you wish to read/write to.

kafka_consume

Double-click on the gauge and slider nodes and define a Dashboard group. Also adjust the labels, range and sizing to meet your requirements.

kafka_gauge

After the logic is complete hit the Deploy button to run the logic. The web dashboard is available at: http://your_node_red_ip:1880/ui.

kafka_phone

Final Comment

I found Node-Red and Kafka to be easy to use in a simple standalone environment. However when I tried to connect to a Cloud based Kafka service (https://www.cloudkarafka.com/) I quickly realized that there is a security component that needs to be defined in Node-Red. Depending on the cloud service that is used some serious testing will probably be required.

 

Small Standalone Databases for IoT

For many IoT projects you want to historically save your data, however if you’re only saving a small amount of data using a database server such as MySQL can be overkill.

I wanted to come up with a way to save my data and be able to port it between projects and hardware.  My goal was to use Python because it has the best Raspberry Pi support. I looked at some options like:

  • CSV files – simple but filtering and sorting isn’t so easy
  • dBase files – good “old school” approach, however it’s no longer support in Excel and MS Access so viewing the data offline is a little ugly
  • Python Pickle and PickleDB – great for saving game scores but no real sorting
  • SQLite – excellent light weight database

For my light weight Python applications I was really happy with TinyDB. TinyDB offers:

  • a JSON storage interface, so you could easily migrate to MongoDB (or equivalent)
  • pure Python with no dependencies,  (so it runs on all OS’s and versions of Python)
  • supports queries and searches

Getting Started with TinyDB

To install TinyDB :

pip install tinydb

TinyDB is standalone so there are no server components. Your Python code talks directly to one of more JSON files, or you can also create multiple tables in one JSON.

Below is a Python example that create a JSON file with two tables.


# Use TinyDB to create a JSON file with 2 tables
#
from tinydb import TinyDB, Query

db = TinyDB('db2.json')
db.purge_tables() # Clear out the file's old data

table = db.table('Moisture')

table.insert({"date":"2019-05-06","time":"16:00","moisture":200})
table.insert({"date":"2019-05-06","time":"16:10","moisture":210})
table.insert({"date":"2019-05-06","time":"16:20","moisture":220})

table2 = db.table('Light')

table2.insert({"date":"2019-06-06","time":"16:00","light":30})
table2.insert({"date":"2019-06-06","time":"16:10","light":35})
table2.insert({"date":"2019-06-06","time":"16:20","light":36})

db.close()

#
import json

with open('db2.json') as f:
data = json.load(f)

print("JSON file with 2 tables\n")
print(json.dumps(data, indent = 4, sort_keys=True))

The JSON file will have a format like:

json_2tables

A Test Example

For a test example I wanted to simulate a month’s worth of 5 minute data. Then I would try some queries to show the data.

 
# Create a month's worth of 10 minute simulated JSON data
#
# Use a format of:
#    "Pi_1": {
#        "1": {
#            "date": "2019-04-01",
#            "time": "01:00",
#            "moisture": 577,
#            "light": 30
#        }

from tinydb import TinyDB, Query
import random

db = TinyDB('/writing/blog/tinydb/db1.json')
db.purge_tables()
table = db.table('Pi_1')

for iday in range(1,31):
    print("day: ",iday)
    for ihour in range(0,24):
        print("\thour: ", ihour)
        for imin in range(0,56,5):
            thedate = "2019-04-{:02}".format(iday)
            thetime = "{:02}:{:02}".format(ihour, imin)
            table.insert({"date": thedate,"time": thetime ,"moisture": random.randint(200,600),"light":  random.randint(20,100)})


print ("Finished updating JSON file\n\n")
db.close(

Once the JSON data file was created I used a simple Python Tkinter application to query the data by day. My goal was to see how easy it was to query the data and return results. Below is a my TinyDB Tkinter viewing application and code. The bottom slider is used to select the day in the month.

dbtiny_data


# Use a Tkinter 
# Create a dialog that uses TinyDB to query 1 day of data
# from a JSON file
#
from tinydb import TinyDB, Query
import tinydb 
import tkinter as tk

db = TinyDB('db1.json')

# get a selected days data
def sel():
   thedate = "2019-04-{:02}".format(int(var.get()))
   print(thedate)
   table = db.table('Pi_1') 
   data1 = table.search(Query()['date'] == thedate)
   print(data1)
   if data_list.size() > 0:
       data_list.delete(1,data_list.size() -1 )
   for items in data1:
       data_list.insert(data_list.size()+ 1,
                        items['date'] + " " + items['time'] +
                        " -  Moisture : " + str(items['moisture']) +
                        " -  Light : " + str(items['light']) )


root = tk.Tk()
root.title("DBtiny - Daily Data")

var = tk.DoubleVar()
data_list = tk.Listbox(root,height=20, width= 60)
data_list.grid(row=1,column=1)
                     
day_scale = tk.Scale( root,variable = var, from_=1,to=30,orient='horizontal',length=300 ).grid(row=2,column=1)

button1 = tk.Button(root, text="Get Daily Values", command=sel).grid(row=3,column=1)

root.mainloop()

Summary

TinyDB is a good solution for small IoT projects where I’m using Python and I’m not collecting a lot of data.

My next step would be to look at presenting the JSON data in Node-Red.

 

Mosquitto MQTT Server

There are a number of excellent open source MQTT (Message Queue Telemetry Transport) brokers or servers that are available. Eclipse Mosquitto is a very popular light weight server that can be run on lower end hardware such as Raspberry Pi modules.

In this blog I’ll look at setting up a Mosquitto broker and I’ll setup some connections to it using testing tools and then use JavaScript with Web Sockets to look at Mosquitto diagnostics.

Getting Started

An MQTT project will have the following components:

  • MQTT Broker. The Mosquitto Server that manages and distributes messages
  • MQTT Publisher. A source for the data, such as a Arduino or Raspberry Pi
  • MQTT Subscriber. An application that wants the data

There is a good selection of MQTT libraries and test utilities for MQTT publishers and subscribers.

To install the Mosquitto broker and test utilities on a Raspberry Pi or Debian system enter:

sudo apt-get update
sudo apt-get install mosquitto mosquitto-clients

For Windows and other systems see the Mosquitto download documentation.

The features of the mosquitto broker are defined by the mosquitto configuration file (/etc/mosquitto/mosquitto.conf). For simple testing you can leave the configuration as is, however if you need to any added features like bridging of MQTT brokers, adding diagnostics or setting up MQTT for web access then you’ll need to edit mosquitto.conf.

For my project I wanted to add diagnostic (with a 10 second update), and I wanted MQTT to also run on Websockets. To do this I edited the config file by:

sudo nano /etc/mosquitto/mosquitto.conf

then I added the lines:

listener 8080
protocol websockets
sys_interval 10

The websockets can be set to any port number (9001, 8000,8080 etc.). After the file has been modified you’ll need to restart the Mosquitto service:

sudo service mosquitto restart

Test Clients

To test that your Mosquitto MQTT broker is running you can open 2 terminal sessions and run the mosquitto_sub and mosquitto_pub utilities.

In the first session we start mosquitto_sub with it subscribing to a topic called mytopic1:

~$ mosquitto_sub -t mytopic1
value1
1234

In the second session we use mosquitto_pub and we can publish some values to the topic mytopic1:

~$ mosquitto_pub -t mytopic1 -m "value1"
~$ mosquitto_pub -t mytopic1 -m 1234

If everything is working correctly then the published values should appear in the subscribing session.

JavaScript, Websockets and  Mosquitto Diagnostics

We can create a simple web page to test out the Javascript, Websockets and Mosquitto Diagnostics.

The Mosquitto Diagnostics have a topic of : $SYS/broker/… Some of these diagnostics include:

$SYS/broker/bytes/received
$SYS/broker/bytes/sent
$SYS/broker/clients/connected
$SYS/broker/clients/maximum
$SYS/broker/clients/total
$SYS/broker/heap/current size
$SYS/broker/heap/maximum size
$SYS/broker/load/connections/+
$SYS/broker/load/bytes/received/+
$SYS/broker/load/bytes/sent/+
$SYS/broker/load/messages/received/+
$SYS/broker/load/messages/sent/+
$SYS/broker/load/publish/dropped/+
$SYS/broker/load/publish/received/+
$SYS/broker/load/publish/sent/+
$SYS/broker/load/sockets/+
$SYS/broker/messages/received
$SYS/broker/messages/sent
$SYS/broker/publish/messages/received
$SYS/broker/publish/messages/sent
$SYS/broker/retained messages/count
$SYS/broker/store/messages/count
$SYS/broker/store/messages/bytes
$SYS/broker/subscriptions/count
$SYS/broker/version

where  + = 1min, 5min or 15min averages

For a full description of the Mosquitto diagnostics see: https://mosquitto.org/man/mosquitto-8.html .

Next we can create a web page where we can subscribe to some of these diagnostic topics. An MQTT javascript library can be remotely called from:

https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.js

This library is very similiar to Python Paho MQTT library, and it supports both subscribe and publish functions.

<html>
<head>
<title> MQTT Subscribe Message</title>
</head>
<script src="https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.js" type="text/javascript"></script>


<script>
// Define some MQTT variables


function sub_mqtt_msg() {
// Send an MQTT message
  var mqtt_server = document.getElementById("mqtt_server").value;
  var mqtt_port = Number(document.getElementById("mqtt_port").value);


  client = new Paho.MQTT.Client(mqtt_server, mqtt_port,"");
  client.onMessageArrived = onMessageArrived;
  client.onMessageArrived = onMessageArrived;
  client.connect({onSuccess:onConnect});
  document.getElementById("submsg").innerHTML = "Trying to connect...<br>";

}
function onConnect() {
  document.getElementById("submsg").innerHTML = "New connection made...<br>";
  var mqtt_destname = document.getElementById("mqtt_destname").value;  
  client.subscribe(mqtt_destname);
  document.getElementById("submsg").innerHTML = "Subscribing to topic: " + mqtt_destname + " ...<br>";
}
function onMessageArrived(message) {
  var result = message.destinationName + " : " + message.payloadString + "<br>";
  document.getElementById("submsg").innerHTML += result;
}

</script>
<body>
<h1>MQTT Subscribe Test Page</h1>

Server Name: <input type="text" id="mqtt_server" value="192.168.0.121"><br><br>
Websocket: <input type="text" id="mqtt_port" value="8080"><br><br>
DestinationName: <input type="text" id="mqtt_destname" value="$SYS/broker/#"><br><br>

<button onclick="sub_mqtt_msg()">Subscript to MQTT</button>
<hr>
<h2>Subscribed Messages:</h2>
<div id=submsg></div>
</body>

</html>

By subscribing to $SYS/broker/# we can get see all the different diagnostic values.

java_ws_diag2

Summary

Mosquitto is an excellent MQTT broker that can be run on both high end servers and low end hardware like Raspberry Pi’s.

It is important to note that with most MQTT libraries, (i.e. Python Paho, JavaScript and Node Red) the default setting is to not retain only values. This can be a problem if a new MQTT subscribers starts because it will not have any data until new values are published. To ensure that old values are available set the RETAIN=YES option on the MQTT publish call.

In this blog I wanted to examine, the diagnostic features of the Mosquitto MQTT broker, however there are many more features and functions of MQTT communications that need to be examined. There are some excellent MQTT guides on the Internet. I tried to document some of my notes on MQTT with Javascript and Charts, and MQTT with Arduino and Node Red.