Bash with MQTT

I’m working on re-purposing an old router. One of my goals is to bring sensor and performance data from the router to a Home Assistant node. The router doesn’t not have a lot of space so I’d prefer to use Bash rather than Python for MQTT communications.

While I was working on the project I wanted to use some simple tools to view the data, unfortunately I wasn’t able to find any information on how to make a good MQTT client in Bash.

This blog documents how I used Bash to show bar charts of MQTT data.

Setup

The first step is to install the Mosquitto command line tools on the OpenWrt router:

# Update the package manager list
opkg update
# Install the MQTT client utiliy
opkg install mosquitto-client-nossl

When I have some time I’d like to look at making some more MQTT Bash scripts that could be used with file input.

The next step is to install the Mosquitto client on my Linux PC and Raspberry Pi:

sudo apt-get install mosquitto-clients

There are a number of MQTT brokers that can be used, both Home Assistant and Node Red have reliable brokers. The Mosquitto broker can also be loaded on a Linux, MacOS and Windows node.

Publish MQTT Data in Bash

The Mosquitto client can be used to both publish and subscribe to MQTT data.

For my router project, I wanted the: temperature (from a USB thermometer), idle time, % used space and available space. Below is the script that passed the data to the mosquitto_pub (publishing) tool:

#!/usr/ash
#
# mqtt_data.sh - send data to MQTT broker
#
# Get Data values
temp=$(/usbstick/temper.sh)
idle=$(vmstat | awk '{ if (NR==3) print $15}')
used=$(df | awk '{if (NR==4) printf "%.f\n", $5 }')
space=$(df | awk '{if (NR==4) printf "%.1f\n", $4/1000 }')

echo "$temp $idle $used $space"

# Publish Data
server="192.168.0.111"
pause=2

mosquitto_pub -h $server -t rtr_temp -m $temp
sleep $pause
mosquitto_pub -h $server -t rtr_idle -m $idle
sleep $pause
mosquitto_pub -h $server -t rtr_used -m $used
sleep $pause
mosquitto_pub -h $server -t rtr_space -m $space

I added a pause between each publish to let me watch the actions.

The cron utility can be used to schedule the running of the script. Once a minute is the fastest time available with cron, so if faster times are needed the script could cycle with a while loop.

Read/Subscribe to MQTT Data

The mosquitto_sub client tool can be used to read or subscribe to the data.

To look at the router points:

$ service="192.168.0.111"
$ mosquitto_sub -h $server -v -t rtr_idle -t rtr_temp -t rtr_used -t rtr_space

rtr_temp 26.25
rtr_idle 100
rtr_used 66
rtr_space 1.1

The -v (–verbose) option show output with the topic names and the values. Topics, -t option, can be put on the command line or passing in as a file.

For single point monitoring the Zenity utility can be used. This utility is typically preloaded on most Linux and Rasp Pi systems. A script to create a progress bar dialog with MQTT data is:

# Send MQTT values to a progress bar
( 
  while :; do
  msg=$(mosquitto_sub -h 192.168.0.111 -C 1 -t rtr_temp) 
  echo $msg
  echo "#$msg  Deg C"
  done
  ) | zenity --progress  --title="Router External Temperature"

Multipoint Progress Dialogs

The Zenity utility can only manage single point progress bars. For multiple point progress bars the YAD (Yet Another Dialog) package can be used.

To install YAD on Raspberry Pi’s and Ubuntu: sudo apt-get install yad

Below is some script that will show multiple points in a YAD dialog:

#!/usr/bash
#
# mqtt_bars.sh - Show multiple MQTT Topics on a Dialog
#
server="192.168.0.111"
topics=("rtr_idle" "rtr_temp" "rtr_used" )
scale=(100 40 100 )
units=( '%' 'degC' '%'  )
title="Router MQTT Points"

#Build topic and yad strings
yadstr=" "
for i in ${topics[@]}; do
  topstr="$topstr -t $i"
  yadstr="$yadstr --bar=$i"
done

echo "Press [CTRL+C] to stop..."
# Cycle thru 1 message at a time to YAD 
(
while : 
do 
  msg=$(mosquitto_sub -h $server -v -C 1 $topstr) 
  IFS=' ' read -a data <<< "$msg" 
  # match returned msg to order of bars, write value/label
  for i in "${!topics[@]} "
  do 
    if [ "${data[0]}" = "${topics[i]}" ]
    then 
      let j=i+1 ; # YAD indices start at 1
      # Rescale bar to defined scale
      barsize=$(bc <<< "${data[1]}*100/${scale[i]}")
      #barsize=100
      echo "$j:$barsize"
      echo  "$j:#${data[1]} ${units[i]}"
  fi
  done
done 
)  | yad --multi-progress $yadstr --title $title

Final Comments

There are probably lots of good 3rd party tools like Gnuplot that could be connected to mosquitto_sub to show real charts.

Mosquitto MQTT Server

There are a number of excellent open source MQTT (Message Queue Telemetry Transport) brokers or servers that are available. Eclipse Mosquitto is a very popular light weight server that can be run on lower end hardware such as Raspberry Pi modules.

In this blog I’ll look at setting up a Mosquitto broker and I’ll setup some connections to it using testing tools and then use JavaScript with Web Sockets to look at Mosquitto diagnostics.

Getting Started

An MQTT project will have the following components:

  • MQTT Broker. The Mosquitto Server that manages and distributes messages
  • MQTT Publisher. A source for the data, such as a Arduino or Raspberry Pi
  • MQTT Subscriber. An application that wants the data

There is a good selection of MQTT libraries and test utilities for MQTT publishers and subscribers.

To install the Mosquitto broker and test utilities on a Raspberry Pi or Debian system enter:

sudo apt-get update
sudo apt-get install mosquitto mosquitto-clients

For Windows and other systems see the Mosquitto download documentation.

The features of the mosquitto broker are defined by the mosquitto configuration file (/etc/mosquitto/mosquitto.conf). For simple testing you can leave the configuration as is, however if you need to any added features like bridging of MQTT brokers, adding diagnostics or setting up MQTT for web access then you’ll need to edit mosquitto.conf.

For my project I wanted to add diagnostic (with a 10 second update), and I wanted MQTT to also run on Websockets. To do this I edited the config file by:

sudo nano /etc/mosquitto/mosquitto.conf

then I added the lines:

listener 8080
protocol websockets
sys_interval 10

The websockets can be set to any port number (9001, 8000,8080 etc.). After the file has been modified you’ll need to restart the Mosquitto service:

sudo service mosquitto restart

Test Clients

To test that your Mosquitto MQTT broker is running you can open 2 terminal sessions and run the mosquitto_sub and mosquitto_pub utilities.

In the first session we start mosquitto_sub with it subscribing to a topic called mytopic1:

~$ mosquitto_sub -t mytopic1
value1
1234

In the second session we use mosquitto_pub and we can publish some values to the topic mytopic1:

~$ mosquitto_pub -t mytopic1 -m "value1"
~$ mosquitto_pub -t mytopic1 -m 1234

If everything is working correctly then the published values should appear in the subscribing session.

JavaScript, Websockets and  Mosquitto Diagnostics

We can create a simple web page to test out the Javascript, Websockets and Mosquitto Diagnostics.

The Mosquitto Diagnostics have a topic of : $SYS/broker/… Some of these diagnostics include:

$SYS/broker/bytes/received
$SYS/broker/bytes/sent
$SYS/broker/clients/connected
$SYS/broker/clients/maximum
$SYS/broker/clients/total
$SYS/broker/heap/current size
$SYS/broker/heap/maximum size
$SYS/broker/load/connections/+
$SYS/broker/load/bytes/received/+
$SYS/broker/load/bytes/sent/+
$SYS/broker/load/messages/received/+
$SYS/broker/load/messages/sent/+
$SYS/broker/load/publish/dropped/+
$SYS/broker/load/publish/received/+
$SYS/broker/load/publish/sent/+
$SYS/broker/load/sockets/+
$SYS/broker/messages/received
$SYS/broker/messages/sent
$SYS/broker/publish/messages/received
$SYS/broker/publish/messages/sent
$SYS/broker/retained messages/count
$SYS/broker/store/messages/count
$SYS/broker/store/messages/bytes
$SYS/broker/subscriptions/count
$SYS/broker/version

where  + = 1min, 5min or 15min averages

For a full description of the Mosquitto diagnostics see: https://mosquitto.org/man/mosquitto-8.html .

Next we can create a web page where we can subscribe to some of these diagnostic topics. An MQTT javascript library can be remotely called from:

https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.js

This library is very similiar to Python Paho MQTT library, and it supports both subscribe and publish functions.

<html>
<head>
<title> MQTT Subscribe Message</title>
</head>
<script src="https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.js" type="text/javascript"></script>


<script>
// Define some MQTT variables


function sub_mqtt_msg() {
// Send an MQTT message
  var mqtt_server = document.getElementById("mqtt_server").value;
  var mqtt_port = Number(document.getElementById("mqtt_port").value);


  client = new Paho.MQTT.Client(mqtt_server, mqtt_port,"");
  client.onMessageArrived = onMessageArrived;
  client.onMessageArrived = onMessageArrived;
  client.connect({onSuccess:onConnect});
  document.getElementById("submsg").innerHTML = "Trying to connect...<br>";

}
function onConnect() {
  document.getElementById("submsg").innerHTML = "New connection made...<br>";
  var mqtt_destname = document.getElementById("mqtt_destname").value;  
  client.subscribe(mqtt_destname);
  document.getElementById("submsg").innerHTML = "Subscribing to topic: " + mqtt_destname + " ...<br>";
}
function onMessageArrived(message) {
  var result = message.destinationName + " : " + message.payloadString + "<br>";
  document.getElementById("submsg").innerHTML += result;
}

</script>
<body>
<h1>MQTT Subscribe Test Page</h1>

Server Name: <input type="text" id="mqtt_server" value="192.168.0.121"><br><br>
Websocket: <input type="text" id="mqtt_port" value="8080"><br><br>
DestinationName: <input type="text" id="mqtt_destname" value="$SYS/broker/#"><br><br>

<button onclick="sub_mqtt_msg()">Subscript to MQTT</button>
<hr>
<h2>Subscribed Messages:</h2>
<div id=submsg></div>
</body>

</html>

By subscribing to $SYS/broker/# we can get see all the different diagnostic values.

java_ws_diag2

Summary

Mosquitto is an excellent MQTT broker that can be run on both high end servers and low end hardware like Raspberry Pi’s.

It is important to note that with most MQTT libraries, (i.e. Python Paho, JavaScript and Node Red) the default setting is to not retain only values. This can be a problem if a new MQTT subscribers starts because it will not have any data until new values are published. To ensure that old values are available set the RETAIN=YES option on the MQTT publish call.

In this blog I wanted to examine, the diagnostic features of the Mosquitto MQTT broker, however there are many more features and functions of MQTT communications that need to be examined. There are some excellent MQTT guides on the Internet. I tried to document some of my notes on MQTT with Javascript and Charts, and MQTT with Arduino and Node Red.