Charts in 1 Line of Bash

There are some great charting packages out there, but for quick tests and playing around it’s nice to keep things simple. I wanted to do some basic Raspberry Pi charting and found that with the Gnuplot utility it took as little as 1 line of Bash to get the job done.

This blog shows some examples, probably the key points are:

  • Data can be piped to Gnuplot
  • Bash While-loops can be used to create real time/dynamic plots

Single Line Chart from a File

Gnuplot plot has a full scripting interface that allows users to create very complex presentations.

Gnuplot script can also be passed on the command line, with the options: -p (persist, spawn chart) and -e (execute command):

$ # Plot a file with 5 points, show file first
$ cat 5pnts.txt
2.7
3.5
4.1
3.9
5.6
$ # Plot the file as a line
$ gnuplot -p -e "plot '5pnts.txt' with lines title '5 Test Points' "

Multi-line Chart

The multi-line example is like the single line chart with the exception that the using statement is called for each plot line, 1:2 is the first line (x-axis column):(y-axis column) , and 1:3 is the second series line.

$ # Show the first 5 line of the data
$ head -n 5 data1.txt, Note: Gnuplot skips row starting with "#"
#t  Angle Error 
0.0	-14.7	3.6
1.0	8.6	    3.6
2.1	28.8	3.0
3.1	46.7	3.4
$
$ # plot, first column is x, next 2 columns are y series
$ gnuplot -p -e "plot 'data1.txt' \
  using 1:2 with lines title 'Angle','data1.txt' \
  using 1:3 with lines title 'Error'"

For-Loop Piped to a Line Chart

Rather than creating a text file, the output from a group of echo statements can be piped to Gnuplot. The ‘<cat’ definition is used for piped input.

$# Plot 10 random points
$ ( for i in `seq 1 10`; do echo "$RANDOM\n";  done )| \
  gnuplot -p -e "plot '<cat' with lines title '10 Random Pts'"

A more complex example would be to add a time on the X-axis and run a calculation at a sampled interval.

# Create a calc function
mycalc() {
  # Show the time and cpu idle time
  thetime=$(date +'%T')
  idle=$(top -n 1 | grep id | awk '{print $8}')
  echo "$thetime $idle\n"
}
# Run the calc function in the for-loop
# Define the y range, x scale as time
( for i in `seq 1 20`; do mycalc; sleep 1;  done )| \
  gnuplot -p -e "set yrange [50:100]; set xdata time; \
  set timefmt '%H:%M:%S';set format x '%H:%M:%S'; \
  plot '<cat' using 1:2 with lines title 'CPU Idle Time' "

Bar Charts from a File

For the bar chart example the plot options are defined in a variable. This example uses the iostat command to get the user and idle time usage. Awk is used to parse the 4th row and then get the individual values.

# Create a file with user and idle CPU usage
# Column: 1=bar position, 2=value, 3=label
iostat | awk '{if(NR==4) print "0 " $1 " user"}' > cpu0.dat
iostat | awk '{if(NR==4) print "1 " $3 " idle" }' >> cpu0.dat

# Define the options as a variable
options="set title 'CPU Diagnostics'; set boxwidth 0.5; \
  set style fill solid; "

# Plot the bars with labels, note: the label offset 
gnuplot -p -e "$options ;plot 'cpu0.dat' \
 using 1:2:xtic(3) with boxes notitle, \
 '' using 1:2:2 with labels font 'Helvetica,15' \
 offset 0.9,0.8  notitle" 

Bars with Dynamic Updates

A Bash while-loop can be used to dynamically get new data and pass it to Gnuplot.

Below is an example that refreshes a bar chart every 5 seconds. The refresh time is shown in the bar series title.

mycalc2() {
  # Show the time and user/cpu idle time
  # note: top gets instantaneous values, iostat uses averages
  top -n 1 | grep %Cpu | awk '{print "0 " $2 " user"}' > cpu0.dat
  top -n 1 | grep %Cpu | awk '{print "1 " $8 " idle"}' >> cpu0.dat
}
# define chart options
options="set title 'CPU Diagnostics'; set boxwidth 0.5; \
  set style fill solid; set yrange [0:100]; set xrange [-0.5:1.5]"

# create an infinite while loop to get data, and then plot
# note1: gnuplot needs a while loop to replot
# note2: use a big pause time in gnuplot or exiting will be tough 
( while :; do mycalc2; sleep 10;  done )| \
gnuplot -p -e "$options ; plot 'cpu0.dat' \
 using 1:2:xtic(3) with boxes \
 title \"$(date '+%T')\" ; while (1) { replot; pause 5 }" 

Summary

Using Gnuplot with the command line option allows for some quick and easy charting. For more detailed work try creating a Gnuplot script file.

Gemini Protocol for Lightweight Internet Apps

The Gemini protocol is a very simple and light Internet protocol.

Unlike HTML files that contain layers of tags, style sheets and Javascript, a Gemini document is a simple readable document.

In this blog I wanted to document:

  • Gemini servers and clients using only 1 line of Bash
  • How to use large ASCII text in Gemini documents
  • How to create simple bar charts
  • How to create Gemini CGI apps in Python

Getting Started

A Gemini document only supports a few statements and graphic images such as JPEG or PNG are not supported.

An example Gemini document with the common formatting options:

# Heading level 1 (H1) 
## Heading level 2 (H2)
### Heading level 3 (H3)

=> testpage.gmi A link to another page.

> This line will show as a block-quote. 

A list of items
* This is the first list item.
* This is another list item.

```
Code or ASCII Block 
   _   ___  ___ ___ ___   _____       _   
  /_\ / __|/ __|_ _|_ _| |_   _|__ __| |_ 
 / _ \\__ \ (__ | | | |    | |/ -_|_-<  _|
/_/ \_\___/\___|___|___|   |_|\___/__/\__|

```

Within a Gemini browser this file would look like:

Content Type

The content type is used by browsers and applications to determine how to manage the requested file.

Typically the content type is managed by server, for example a web server will send a HTTP/1.0 200 OK prior to sending the HTML file.

For the Gemini protocol the content type is: 20 text/gemini . Depending on the Gemini server the user may need to be add the content type manually. (More about this in Bash and CGI servers).

Simple Bash Gemini Servers and Clients

For basic testing a one line Bash statement can be used for custom Gemini servers and clients.

The Gemini protocol typically uses SSL (Secure Sockets Layer) and TLS (Transport Layer Security) encryption so the Bash ncat utility is needed (Note: the nc command does not support SSL).

Below is an example of single Gemini request:

The Gemini server is defined by using the -l , listen option. When a client requests data, the cat statement is used with a pipe (|) to output the file testpage.gmi.

The Gemini client echo’s a “from PC” message with its request, this helps identify which client is requesting data.

A simple Bash ncat statement doesn’t manage the content type so a “20 text/gemini” line is added to the top of the test page.

Dynamic Bash Data

In the earlier example the Bash server is only supporting 1 request then exiting.

A while loop can be added to pass a Bash script to the ncat statement. Below is an example of Bash script (showdata.sh) that show CPU data using the vmstat utility:

#!/bin/bash
#
# showdata.sh - Output data for Gemini Bash Server 
#
echo "20 text/gemini"
echo  " "
echo "#VMSTAT"
echo  " "
date +"%T"
echo  " "
# set Gemini formating for ASCII
echo  "\`\`\`"
#vmstat
top -n 1
echo  "\`\`\`"

To make the script executable use the command: chmod +x showdata.sh

The Bash command to run this script as a Gemini server is:

while true; do ./showdata.sh | ncat  -l -p 1965 --ssl; done

The earlier Bash Gemini client command can be used, or a Gemini browser/client app can be used. The handling of SSL/TLS encryption will vary with the client app. I used the very basic Zain app (https://gitgud.io/sathariel/zain) :

(Note: for the Zain client I needed to load tcl/tls, sudo apt-get install -y tcl-tls)

Using a 1 line Bash Gemini server is great for basic testing but I wouldn’t recommend if you want to connect to variety of different Gemini client applications.

Large ASCII Text

Gemini documents don’t support different font sizes, a workaround is to use the figlet tool to generate multi-line text strings. Figlet is installed on Ubuntu/Debian/Raspbian by:

sudo apt install figlet

Figlet has a number of font styles that use 2-5 line height characters:

When using ASCII headings the Gemini code formatting option should be used, and this has three backticks (“`) before and after the headings.

The earlier example can be modified to have ASCII headings:

#!/bin/bash
#
# showdata2.sh - Output Large Headings to a Gemini Bash Server 
#
echo "20 text/gemini"
echo  " "
echo  "\`\`\`"
# Generate large text 
figlet -f standard "Raspberry Pi"
figlet -f small -m 2  $(date +"%T")
# show CPU stats
vmstat
echo  "\`\`\`"

Bar Charts

Horizontal ASCII bar charts can be created by using the printf statement with different ASCII fill characters. For example:

# show a label with bar and value text 
#
label="temp"; val=20;
bar="printf '█%.0s' {1..$val}" ; 
printf '\n%-5s ' $label; eval $bar
printf '░%.0s' {1..5} ;
printf ' 50 C\n'

temp  ████████████████████░░░░░ 50 C

This bar logic can be using in a Raspberry Pi Stats page that looks at idle time and free space on the SD card:

#!/bin/bash
#
# pi_stats.sh - Output Pi Stats as a Gemini Page
#
echo "20 text/gemini"
echo  " "
# Put the rest of the Gemini document into code block mode
echo  "\`\`\`"
# Generate large text 
figlet -f standard "Pi Stats"

# Show the time
echo "Time: $(date +'%T')"
echo ""

# Get idle time, scale 0-50
idle=$(top -n 1 | grep id | awk '{print $8}')
barsize=$(echo $idle | awk '{printf "%0.f" , $1/2}')
thebar="printf '█%.0s' {1..$barsize}"
graysize=$(expr 50 - $barsize)
thegray="printf '░%.0s' {1..$graysize}"
printf 'Idle Time  '; eval $thebar; eval $thegray ; echo " $idle %"
echo  ""

# Get free space on SD card, scale 0-50
freesp=$(df | grep root | awk '{printf "%0.f", $5/2}')
barsize=$(echo $freesp | awk '{printf "%0.f" , $1/2}')
thebar="printf '█%.0s' {1..$barsize}"
graysize=$(expr 50 - $barsize)
thegray="printf '░%.0s' {1..$graysize}"
printf 'Free Space '; eval $thebar; eval $thegray ; echo " $freesp %"

echo  "\`\`\`"

To run this page use

while true; do ./pi_stats.sh | ncat  -l -p 1965 --ssl; done

Python CGI Pages

There are a number of good Gemini servers, for my testing I used the Python based Jetforce server, it is installed by:

pip install jetforce

To run the Jetforce server it is important to define a home directory, the allowable hosts that can connect (0.0.0.0 is all IP4 nodes) and the server’s hostname:

# Start the jetforce Gemini server for all IP4 hosts
jetforce --dir /home/pi/temp --host "0.0.0.0" --hostname 192.168.0.105 &
# Start jetforce without hardcoding hostname
# jetforce --dir /home/pi/temp --host "0.0.0.0" --hostname $(hostname -I) &

By default CGI (Common Gateway Interface) files are defined in the directory cgi-bin which is under the home directory.

Unlike a one-line Bash server, Jetforce server will pass environment variables like QUERY_STRING and host and remote connection information.

CGI programs can be written in a variety of programming languages. For this example I wanted to pass Raspberry Pi BME280 temperature, pressure and humidity sensor information to a Gemini CGI page.

The CGI program was written in Python and I installed a bme280 and figlet library:

pip install RPi.bme280
pip install pyfiglet

The Python script (sersor2gmi.py) outputs a Gemini content type, a Figlet title and then the sensor data values:

#!//usr/bin/python3
#
# sersor2gmi.py - send BME280 sensor data to a Gemini page
#
import smbus2
import bme280
import pyfiglet

# find device address via: i2cdetect -y 1
port = 1
address = 0x77
bus = smbus2.SMBus(port)

calibration_params = bme280.load_calibration_params(bus, address)

# the sample method returns a compensated_reading object
data = bme280.sample(bus, address, calibration_params)

# Output a Gemini page
print("20 text/gemini") #Note: Some Gemini CGI servers may do this
print("```") # use code mode
#print(pyfiglet.figlet_format("Pi BME280 Data", font = "small"))
print(pyfiglet.figlet_format("Pi BME 280 Data", font = "small"))

print("Temperature:" + "{:5.1f}".format(data.temperature) + " C" )
print("Pressure:   " + "{:5.1f}".format(data.pressure) + " kPa" )
print("Humidity:   " + "{:5.1f}".format(data.humidity) + " %" )
print("```") 

This file was added to the cgi-bin directory and it is made executeable (chmod +x sersor2gmi.py).

Below is the output seen in the Lagrange Gemini browser:

Final Comments

There are a variety of Gemini browsers for Windows, Android, Mac and Linux so if you’re looking for a quick and dirty internet solution Gemini might be a good solution.

I like how Gemini documents are totally readable, I can’t say the same for most web pages.

The one thing that I missed with Gemini pages is the ability of show nice charts, text based bars work for simple stuff but doing text based line charts is a little too ugly for me.

Memcached: fast lightweight network cache

For many projects an SQL database is overkill for simple storage of values. As an alternate solution there are a number of excellent distributed in-memory caching systems that can be used.

Two of the most popular in-memory caching systems are Redis and Memcached.

I’m big fan of Redis, and I’ve enjoyed doing projects with it. Redis offer some awesome speed with a small footprint, and it has many features that make it even superior to a messaging system like MQTT.

However if you’re looking for something 100% dead simple you should take a look a memcached, it has a super simple setup and with only a dozen commands so you’ll get up and running in no time.

There are API’s in all the common programming languages. It only takes 1 line of Bash to read or write to memchached.

Getting Started

Memached can be installed on all major OS’s. To install it on Ubuntu/Raspberry Pi:

sudo apt-get install memcached

If you’re using Docker there are some lightweight memcached images (89MB) that can be used. To run the memcached docker image the –net host option should be used:

$ sudo docker run -it --net host memcached

The -it (interactive) option could be useful if there are any errors kicked out.

See the man pages for a full description of memcached options.

Telnet can be used to enter manual enter commands (port 11211 is the default):

$ telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
version
VERSION 1.6.12

lru_crawler metadump all
key=mynum exp=-1 la=1642363105 cas=2 fetch=no cls=1 size=66
END
quit
Connection closed by foreign host.

For more information on the commands and Telnet.

Bash Interface to Memcached

There are lot of help guides for other Python, PHP, GO etc., but not a lot on how to use Bash.

The Bash nc command can be used to read and write to socket. The -q option will close the socket after 0 seconds (after the echo command is sent). The default port is 11211 but this can be changed along with adding some security options. The first example is to use the stats command:

$ echo "stats" | nc -q 0  192.168.0.120 11211 
STAT pid 1
STAT uptime 722
STAT time 1642359428
STAT version 1.6.12
...
END

$ # Find the number of current key items
$ echo "stats" | nc -q 0  192.168.0.120 11211 | grep curr_items
STAT curr_items 1

Set/Get a Key and Value

To set an in-memory key-value store, the syntax is:

set mykey <flags> <ttl> <size>
value

The flags option is typical set to 0. The ttl “time to live” is in seconds, a ttl of 0 is indefinite.The size of the value also needs to be define. (This is taken care of in the Python, PHP… libraries). It’s important to note that a newline (\r\n) is required before the value.

To set a key to a variable mynum with a value of 55 and a indefinite time to live:

$ # hard code the arguments in a set key command
$ # Note: echo -e is used to pass the \r\n for new lines
$ echo -e "set mynum 0 0 2 \r\n55\r" | nc -q 0  127.0.0.1 11211
STORED

A more flexible approach would be to pass variables:

$ # Set a new key/value pair, with no flags and indefinite timeout
$ ipp="192.168.0.120 11211"
$ thekey="mykey1"
$ thenum=55
$ numsize=${#thenum}
$ # send command 
$ echo -e "set $thekey 0 0 $numsize\r\n$thenum\r" | nc -q 0  $ipp
STORED

Get a key/value

The get key command returns 3 lines with the value being the 2nd line. Some awk code can be used to parse out just the value.

$ # Get a new key/value pair
$ ipp="192.168.0.120 11211"
$ thekey="mykey1"
$ 
$ echo "get $thekey" | nc -q 0  $ipp
VALUE mykey1 0 2
60
END
$ # Get just the 2nd line with the value
$ echo "get $thekey" | nc -q 0  $ipp | awk '{if (NR == 2) print $0}'
55
$ 
$ # Store the result in a variable
$ mykey1=$(echo "get $thekey" | nc -q 0  $ipp | awk '{if (NR == 2) print $0}')
$ echo "mykey1 = $mykey1"
mykey1 = 55

Increment/Decrement a Value

The inc / decr commands will increase or decrease a numeric stored key value by a defined amount:

$ #incr/decr a value's number
#
ipp="192.168.0.120 11211"
thekey="mykey1"

# Get starting value
echo "get mynum" | nc -q 0 $ipp | awk '{if (NR == 2) print $0}'
55

thediff=10; # increase the value (only positive values)
# incr and show the value
echo "incr $thekey $thediff" | nc -q 0  $ipp
65

thediff=5; # decrease the value (only positive values)
# incr and show the value
echo "decr $thekey $thediff" | nc -q 0 $ipp
60

Prepending and Appending

Memcached does not have queue or list functionality, if you need this take a look at Redis.

Below is an example script that creates a diagnostic log in a key/value. The append command adds msg text to the end of the overall string.

#!/usr/bin/bash
#
# diagmsg.sh - append msgs to a memcached variable string
#
ipp="192.168.0.120 11211"
msg="1:00 - Base Software Loaded\r"
size=${#msg}

echo -e "set mymsg 0 0 $size\r\n$msg\n\r" | nc -q 0  $ipp

# Create an array of diagnostics
diagmsgs=("2:00 - System Started\r" "2:15 - Getting Data\r" "2:30 - Backing up\r") 
# Append array to value text
for msg in "${diagmsgs[@]}"
do
  size=${#msg}
  #echo "Size: $size $msg";
  echo -e "append mymsg 0 0 $size\r\n$msg\n\r" | nc -q 0  $ipp;
done

# Show the result
echo -e "\nDiagnostic Results\n" 
echo "get mymsg" | nc -q 0 $ipp 

The results of this script would be:

$ bash diagmsg.sh
STORED
STORED
STORED
STORED

Diagnostic Results

VALUE mymsg 0 92
1:00 - Base Software Loaded
2:00 - System Started
2:15 - Getting Data
2:30 - Backing up

END

Final Thought

I played with the Python and PHP library and they were quite easy to use.

By adding a Bash interface to memcache it allows me to use programs like Octave/Matlib where a native interface isn’t available (just use the System call and pass the Bash code).

Sample Code: PHP

PHP example:

<?php
// memc.php - test memcached
//
// Load the PHP memcached component:
//
//    sudo apt-get install php-memcached

$memcache = new Memcached();
$memcache->addServer('localhost', 11211) or die ("Could not connect");

$version = $memcache->getVersion();
var_dump( $version);

$tmp_object = new stdClass;
$tmp_object->str_attr = 'test';
$tmp_object->int_attr = 123;

$memcache->set('key', $tmp_object);
echo "Store data in the cache (data will expire in 10 seconds)<br/>\n";

$get_result = $memcache->get('key');
echo "Data from the cache:<br/>\n";

var_dump($get_result);

?>

Sample Code: Python

Python example:

$ # Install Python memcache library
$ python3 -m pip install pymemcache

$ # Example set/get
$ python3
Python 3.8.10 (default, Nov 26 2021, 20:14:08) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pymemcache.client import base
>>> client = base.Client(('localhost', 11211))
>>> client.set('mykey','teststring')
True
>>> client.get('mykey')
b'teststring'
>>> client.set('mynum',44)
True
>>> client.get('mynum')
b'44'

Gnuplot Speedometer Gauge

Gnuplot is a great package that allows you to do charting from the command line.

Gnuplot has a wide range of plot types but unfortunately a speedometer gauge is not one of them.

This blog documents my notes in creating a dynamic gauge.

Some useful takeaways:

  • A gnuplot can be created with only graphic elements (no chart)
  • A named object can be re-positioned without redrawing the background
  • A gnuplot script can be called like a Bash or Python script with the first line being: #!/usr/bin/gnuplot

A Dynamic Gauge

The script gnuplot.gp is an example that refreshes a gauge chart every 5 seconds with the processor’s idle time. The idle time is obtained using the Linux top command (with some awk to get the 8th line item).

#!/usr/bin/gnuplot
#
# gnuplot.gp - speedometer dial with title as the description/value
#
set xrange [-1:1]
set yrange [0:1]
set angles degrees
set size ratio -1
# r1 = annulus outer radius, r2 = annulus inner radius
r1=1.0
r2=0.5
unset border; unset tics; unset key; unset raxis

set style fill solid noborder

# define a "needle" pointer as object 1
set object 1 circle at first 0,0 front size r1 arc [181:182]  fillcolor rgb 'black'

# define the gauge background
set object circle at first 0,0  size r1 arc [0:180]  fillcolor rgb 'green'
set object circle at first 0,0  size r1 arc [0:10]  fillcolor rgb 'red'
set object circle at first 0,0  size r1 arc [10:20]  fillcolor rgb 'yellow'
set object circle at first 0,0 front size r2 arc [0:180] fillcolor rgb 'black'

# plot the static background 
plot -10 notitle

# Define a partial heading  
variable = "Idle Time\n"
unit = "%"

# Refresh the plot with a new dial setting
while (1) {
  # Get the idle time using the top utility
  idle = system("top -n 1 | grep Cpu | awk '{print $8}'")
  # scale the value from 0-100 to 180-0 (Note: arc starts on the right)
  value = (100 - real(idle)) * 1.8
  # show the value in the title
  set title sprintf('%s %s %s', variable, idle, unit)  font "Ariel,28"
  # reposition the value in the gauge
  set object 1 circle at first 0,0 front size r1 arc [value:(value+2)]  fillcolor rgb 'black'
  replot
  pause 5 
}

The script can be run either from gnuplot or like a Bash script:

$ # run script from gnuplot
$ gnuplot -c gauge.gp
Use Control-C to exit ...
^C
$ # make script executable
$ chmod +x gauge.gp
$ # run script like a Bash script
$ ./gauge.gp
Use Control-C to exit ...
^C

Next Steps…

From here the next steps could be to add command line arguments to make the script more generic, such as:

  • pass the calculation to run (eg. pass the bash top command)
  • pass the scaling, title and units

Bash with MQTT

I’m working on re-purposing an old router. One of my goals is to bring sensor and performance data from the router to a Home Assistant node. The router doesn’t not have a lot of space so I’d prefer to use Bash rather than Python for MQTT communications.

While I was working on the project I wanted to use some simple tools to view the data, unfortunately I wasn’t able to find any information on how to make a good MQTT client in Bash.

This blog documents how I used Bash to show bar charts of MQTT data.

Setup

The first step is to install the Mosquitto command line tools on the OpenWrt router:

# Update the package manager list
opkg update
# Install the MQTT client utiliy
opkg install mosquitto-client-nossl

When I have some time I’d like to look at making some more MQTT Bash scripts that could be used with file input.

The next step is to install the Mosquitto client on my Linux PC and Raspberry Pi:

sudo apt-get install mosquitto-clients

There are a number of MQTT brokers that can be used, both Home Assistant and Node Red have reliable brokers. The Mosquitto broker can also be loaded on a Linux, MacOS and Windows node.

Publish MQTT Data in Bash

The Mosquitto client can be used to both publish and subscribe to MQTT data.

For my router project, I wanted the: temperature (from a USB thermometer), idle time, % used space and available space. Below is the script that passed the data to the mosquitto_pub (publishing) tool:

#!/usr/ash
#
# mqtt_data.sh - send data to MQTT broker
#
# Get Data values
temp=$(/usbstick/temper.sh)
idle=$(vmstat | awk '{ if (NR==3) print $15}')
used=$(df | awk '{if (NR==4) printf "%.f\n", $5 }')
space=$(df | awk '{if (NR==4) printf "%.1f\n", $4/1000 }')

echo "$temp $idle $used $space"

# Publish Data
server="192.168.0.111"
pause=2

mosquitto_pub -h $server -t rtr_temp -m $temp
sleep $pause
mosquitto_pub -h $server -t rtr_idle -m $idle
sleep $pause
mosquitto_pub -h $server -t rtr_used -m $used
sleep $pause
mosquitto_pub -h $server -t rtr_space -m $space

I added a pause between each publish to let me watch the actions.

The cron utility can be used to schedule the running of the script. Once a minute is the fastest time available with cron, so if faster times are needed the script could cycle with a while loop.

Read/Subscribe to MQTT Data

The mosquitto_sub client tool can be used to read or subscribe to the data.

To look at the router points:

$ service="192.168.0.111"
$ mosquitto_sub -h $server -v -t rtr_idle -t rtr_temp -t rtr_used -t rtr_space

rtr_temp 26.25
rtr_idle 100
rtr_used 66
rtr_space 1.1

The -v (–verbose) option show output with the topic names and the values. Topics, -t option, can be put on the command line or passing in as a file.

For single point monitoring the Zenity utility can be used. This utility is typically preloaded on most Linux and Rasp Pi systems. A script to create a progress bar dialog with MQTT data is:

# Send MQTT values to a progress bar
( 
  while :; do
  msg=$(mosquitto_sub -h 192.168.0.111 -C 1 -t rtr_temp) 
  echo $msg
  echo "#$msg  Deg C"
  done
  ) | zenity --progress  --title="Router External Temperature"

Multipoint Progress Dialogs

The Zenity utility can only manage single point progress bars. For multiple point progress bars the YAD (Yet Another Dialog) package can be used.

To install YAD on Raspberry Pi’s and Ubuntu: sudo apt-get install yad

Below is some script that will show multiple points in a YAD dialog:

#!/usr/bash
#
# mqtt_bars.sh - Show multiple MQTT Topics on a Dialog
#
server="192.168.0.111"
topics=("rtr_idle" "rtr_temp" "rtr_used" )
scale=(100 40 100 )
units=( '%' 'degC' '%'  )
title="Router MQTT Points"

#Build topic and yad strings
yadstr=" "
for i in ${topics[@]}; do
  topstr="$topstr -t $i"
  yadstr="$yadstr --bar=$i"
done

echo "Press [CTRL+C] to stop..."
# Cycle thru 1 message at a time to YAD 
(
while : 
do 
  msg=$(mosquitto_sub -h $server -v -C 1 $topstr) 
  IFS=' ' read -a data <<< "$msg" 
  # match returned msg to order of bars, write value/label
  for i in "${!topics[@]} "
  do 
    if [ "${data[0]}" = "${topics[i]}" ]
    then 
      let j=i+1 ; # YAD indices start at 1
      # Rescale bar to defined scale
      barsize=$(bc <<< "${data[1]}*100/${scale[i]}")
      #barsize=100
      echo "$j:$barsize"
      echo  "$j:#${data[1]} ${units[i]}"
  fi
  done
done 
)  | yad --multi-progress $yadstr --title $title

Final Comments

There are probably lots of good 3rd party tools like Gnuplot that could be connected to mosquitto_sub to show real charts.

Bash Bar Charts for Text and Web Pages

This blog documents my notes on creating Bash text bar charts. The key topics are:

  • Create a simple horizontal bars (3 lines)
  • Create dynamic bars (~25 lines for an array of data values)
  • Add support for piped and command line data
  • Include options for: help, resizing, titles
  • Add colour and HTML support

A Horizontal Bar

To create a static bar an printf statements can be used. The seq {0..10} can be used to repeat an ASCII █ fill character 10 times.

$ printf 'LABEL: ' ; \
  printf '█%.0s' {1..10} ; \
  printf ' 10\n'

LABEL: ████████████████████ 10

Unfortunately the printf statement has some limitations on variable substitution. A simple workaround is to create a string and then eval it:

$ label="temp"; val=20;
$ bar="printf '█%.0s' {1..$val}" ; 
$ printf '\n%-5s ' $label; eval $bar ; printf ' %d\n' $val

temp  ████████████████████ 20

Coloured Bars

The tput setaf command can change the foreground, and tput setab is used for background colours. Colour codes are:

tput setab [1-7] # Set the background colour using ANSI escape
tput setaf [1-7] # Set the foreground colour using ANSI escape

Num  Colour    #define         R G B

0    black     COLOR_BLACK     0,0,0
1    red       COLOR_RED       1,0,0
2    green     COLOR_GREEN     0,1,0
3    yellow    COLOR_YELLOW    1,1,0
4    blue      COLOR_BLUE      0,0,1
5    magenta   COLOR_MAGENTA   1,0,1
6    cyan      COLOR_CYAN      0,1,1
7    white     COLOR_WHITE     1,1,1

To reset colours back to the defaults use: tput sgr0

An example to print a red bar and a stack of bars:

$ printf '\nLABEL: ' ; \
   tput setaf 1 ;\
   printf '█%.0s' {1..10} ; \
   printf ' 10\n'

LABEL: ██████████ 10
$ printf '\n 3 Stacked Bars: ' ; \
   tput setaf 1 ;\
   printf '█%.0s' {1..10} ; \
   tput setaf 2 ;\
   printf '█%.0s' {1..8} ; \
   tput setaf 4 ;\
   printf '█%.0s' {1..3} ; \
   printf ' 10+8+3=21\n'

 3 Stacked Bars: █████████████████████ 10+8+3=21

Dynamic Bars

The next step is to create a script that dynamically updates the bars. The tput clear command will clear the terminal screen keep the data and bars in the same location. The script below will dynamically show the CPU temperature, idle time and 2 random values with a 10 second update time.

#!/bin/bash
# 
# cpu_bars.sh - Show new data every 10 seconds
#
while :; do
    # Get data values
    CPUtemp=$(sensors | grep CPU | awk '{print substr($2,2,4)}')
    CPUidle=$(iostat | awk '{if (NR==4) print $6}')
    Random1=$((1+ $RANDOM % 100))
    Random2=$((1+ $RANDOM % 100))

    labels=( CPUtemp CPUidle Random1 Random2)
    values=( $CPUtemp $CPUidle $Random1 $Random2)
    units=( degC % psi mm)

    # Show a title
    tput clear
    printf " %10s " "" 
    tput setaf 7; tput smul;
    printf "%s\n\n" "Show CPU Data ($(date +%T'))"
    tput rmul;

    # cycle thru data and show a label, 
    for index in "${!labels[@]}"
    do
          tput setaf $(expr $index + 1); # don't use 0 (black) 
          printf " %10s " "${labels[index]}"
          eval "printf '█%.0s' {1..${values[index]%.*}}"
          printf " %s %s\n\n" ${values[index]} ${units[index]}
    done
    sleep 10
done

This script is run by: bash cpu_bars.sh .Typical output is below.

Bars in HTML

The ANSI colours are not supported in HTML, so instead HTML tags and style properties can be used.

It is important to use <pre> tags for Bash text output. Code to create two static bars in HTML would be:

$ (printf "<h1>A Bar from Bash</h1>\n" 
 printf "<pre><span style='font-size:24px;color:red'}>\n"
 printf 'LABEL1: ' ; printf '█%.0s' {1..10} ; printf ' 10\n'
 printf "</pre></span>\n") > bar1.htm

$ cat bar1.htm
<h1>A Bar from Bash</h1>
<pre><span style='font-size:24px;color:red'}>
LABEL1: ██████████ 10
</pre></span>

The script cpu_webbars.sh creates HTML output for an array of calculated values:

#!/bin/bash
# 
# cpu_webbars.sh - Show bars in HTML
#

# Get data values
CPUtemp=$(sensors | grep CPU | awk '{print substr($2,2,4)}')
CPUidle=$(iostat | awk '{if (NR==4) print $6}')
Random1=$((1+ $RANDOM % 100))
Random2=$((1+ $RANDOM % 100))

labels=( CPUtemp CPUidle Random1 Random2)
values=( $CPUtemp $CPUidle $Random1 $Random2)
units=( degC % psi mm)
colors=(red blue green magenta)

# Show a title
printf "<h1><center>Show CPU Data ($(date '+%T'))</center></h1>\n"
# cycle thru data and show a label, 
for index in "${!labels[@]}"
  do
  printf "<pre><span style='font-size:18px;color: ${colors[index]} '}>\n"
  printf " %10s " "${labels[index]}"
  eval "printf '█%.0s' {1..${values[index]%.*}}"
  printf " %s %s\n\n" ${values[index]} ${units[index]}
  printf "</pre></span>\n"
done

This script can be run and outputted to an file: bash cpu_webbars.sh > test.htm

Once the basic HTML output is working its possible to add headers and footers to make a more complete page:

header.htm > test.htm ; \
cat cpu_webbars.sh >> test.htm ; \
cat footer >> test.htm 

Piping Data to Bars

A script (hbar0.sh) will read the piped data and then create an array (data) of labels and values. The data array is cycled through and labels and bars are shown with a randomized colour:

#!/bin/bash
# hbar0.sh - Read in piped data  and plot bars
#     format: label,value;label2,value2;  and plot bars
#
input=$(< /dev/stdin) ; # read piped data
# remove new lines in files, and check for ";" after data pair 
input=$(echo $input | tr -d '\n')
IFS=';' read -r -a data <<< $input
printf "\n" 
for element in "${data[@]}"pete@lubuntu:~/Writing/Blog/text_bars
do 
  # make at array of each data element
  IFS=',' read -r -a datapt <<< $element
  # add a random color
  tput setaf $((1+ $RANDOM % 7))
  # print the label, bar and value
  printf " %10s " "${datapt##*[0]}"
  bar="printf '█%.0s' {1..${datapt[1]}}"
  eval $bar
  printf " %s\n\n" ${datapt[1]} 
  tput rmso ; # exit color mode   
done

The script can be tested with piped data:

$ echo "temp,33;pressure,44" | bash hbar0.sh

 temp       █████████████████████████████████ 33 

 pressure   ████████████████████████████████████████████ 44 

A data file can also be passed in using the cat command:

$ cat data0.txt 
temp,44;
humidity,33;
pressure,15;
wind spd,33;
wave ht,3;
$ cat data0.txt | bash hbar0.sh

       temp ████████████████████████████████████████████ 44

   humidity █████████████████████████████████ 33

   pressure ███████████████ 15

   wind spd █████████████████████████████████ 33

    wave ht ███ 3

Removing ANSI Colors

Terminal applications use ANSI color codes which unfortunately is not support on Web pages.

ANSI color codes can be removed from files and strings by:

# Strip out ANSI color codes:
cat infile | sed 's/\x1b\[[0-9;]*m//g' > outfile
$ echo "temp,33;pressure,44" | bash hbar0.sh > hbar0.txt
$ cat hbar0.txt

       temp █████████████████████████████████ 33

   pressure ████████████████████████████████████████████ 44

$ cat hbar0.txt | sed 's/\x1b\[[0-9;]*m//g'

       temp █████████████████████████████████ 33

   pressure ████████████████████████████████████████████ 44

A Final App

With the basics in place I was able to create an app that would support scaling, titles, units, custom colour and web output:

$ ./hbars
usage: hbars [data] [option] 
  -h --help     print this usage and exit
  -c --color    set color to all bars (default 7=white)
                 (0-grey,1-red,2=green,3=yellow,4=blue,5=magenta,6=cyan,7=white)
  -p --pretty   add different colors to bars (-c overrides)
  -t --title    top title
  -w --width    max width of bar (default 50)
  -W --Web      make output HTML formatted
  -f --fontsize fontsize for web output (default 24)

 examples:
   echo 'temp,33,C;pressure,14,psi' | ./hbars -t Weather -p -w 40 
   ./hbars -t Weather -p -w 40  'temp,33;pressure,14' 
   cat data.txt | ./hbars -W -f 24 -t 'Raspi Data' > data.htm

The code:

#!/bin/bash
#
# hbars.sh - show some text bars
#   pass data as:  label1,value1,unit1,color1; label2,value2,unit2,colour2; ....  
#
width=50
title=""
color=7
pretty=False
web=False
font=24

usage() { 
  echo "usage: hbars [data] [option] "
  echo "  -h --help     print this usage and exit"
  echo "  -c --color    set color to all bars (default 7=white)"
  echo "                 (0-grey,1-red,2=green,3=yellow,4=blue,5=magenta,6=cyan,7=white)"
  echo "  -p --pretty   add different colors to bars (-c overrides)"
  echo "  -t --title    top title"
  echo "  -w --width    max width of bar (default 50)"
  echo "  -W --Web      make output HTML formatted"
  echo "  -f --fontsize fontsize for web output (default 24)"
  echo ""
  echo " examples:"
  echo "   echo 'temp,33,C;pressure,14,psi' | ./hbars -t Weather -p -w 40 "
  echo "   ./hbars -t Weather -p -w 40  'temp,33;pressure,14' "
  echo "   cat data.txt | ./hbars -W -f 24 -t 'Raspi Data' > data.htm"
  echo ""

  exit 0
}
# Show help usage if no pipe data and no cmd line data
if [ -t 0 ]  && [ $# -eq 0 ] ; then
  usage
fi
# Check for command line options
while getopts "hpc:t:w:Wf:" arg; do
  case "$arg" in
    h) usage ;;
    c)  color=$OPTARG ;;
    p)  pretty=True; icolor=0 ;;
    t)  title=$OPTARG ;;
    w)  width=$OPTARG ;;
    W)  web=True;;
    f)  font=$OPTARG ;;
  esac
done
#------------------------------------------------
# Setup formatting for text, Web and color
# -----------------------------------------------
if [[ ${color} != 7 && ${pretty} = True ]]; then
  pretty=False
fi
colidx=0

setcolors() {
if [ $web = True ]; then
  colors=(gray red green yellow blue magenta cyan white)
  titlebold="echo '<h1>'"
  titlereset="echo '</h1>'"
  #color_set='echo "<span style=font-size:$(font)px  >" ' 
  #color_set="printf '<span  style=\"font-size:$(font)px;color:${colors[colidx]}\" >'" 
  color_set="printf '<pre><span style=\"color:${colors[colidx]} ; font-size:${font}px \" >'" 
  color_rs="echo '</span></pre>'"
else
  colors=(0 1 2 3 4 5 6 7 )
  titlebold="tput bold; tput smul"
  titlereset="tput rmul; tput rmso"
  color_set="tput setaf ${colors[colidx]}"
  color_rs="tput rmso"
fi
}
setcolors
#----------------------------------------------
# Get data, check if stdlin, file, if not assume string
#----------------------------------------------
if [ -p /dev/stdin ]; then
        lastarg=$(< /dev/stdin)
else
	lastarg=$(echo "${@: -1}")
	if test -f "$lastarg"; then
	  lastarg=$(<$lastarg)
	fi
fi
# Cleanup the input data
lastarg=$(echo $lastarg | sed 's/\n/;/g; s/  / /g; s/\t//g; s/;;/;/g')
IFS=';' read -r -a array <<< $lastarg

# ensure that there is some data
if [[ ${array} == 0 ]]; then
  echo "No data found"
  exit 0
fi
echo "input:$lastarg"
#exit 0
#------------------------------------
# Get max value and max label length
#------------------------------------
maxval=0
maxlbl=10
#echo "array:${array[@]}"
for element in "${array[@]}"
do 
  IFS=',' read -r -a datapt <<< $element
  if (( $(echo "$maxval < ${datapt[1]}" |bc -l) )); then
	maxval=${datapt[1]}
  fi
  if (( $(echo "$maxlbl < ${#datapt[0]}" |bc -l) )); then
	maxlbl=${#datapt[0]}
  fi
done
#---------------------------------
# Print Title - use bold/underline
#---------------------------------
if [[ ! -z $title ]]; then
  printf "\n %${maxlbl}s " " "
  eval $titlebold
  printf "%s" "${title}" ; printf "\n\n"
  eval $titlereset
fi
#------------------------------------
# Cycle thru data and build bar chart
#------------------------------------
for element in "${array[@]}"
do
# check data values
  IFS=',' read -r -a datapt <<< $element
  # check for empty records
  if [ ${#datapt[0]} = 0 ]; then 
    break
  fi
  label=${datapt[0]}
  if [[ ${label} != "-t*" ]]; then 
	  val=${datapt[1]}
	  sval=$(bc <<< "$width * $val / $maxval")

	  # add color, use 4th item if available
	  if [[ ${#datapt[@]} > 3 && $pretty = False ]]; then
		icolor=${datapt[3]}
	  fi
	  if [[ $pretty = True ]] ; then
		let colidx++
		if [ $colidx -gt 7 ]; then
		  let colidx=1
		fi
	  elif [[ ${#datapt[@]} > 3 ]]; then
		colidx=${datapt[3]}
	  else
	  	colidx=$color
	  fi
	  setcolors
          eval $color_set
          printf " %${maxlbl}s " "$label"
	  bar="printf '█%.0s' {1..$sval}"
	  eval $bar; 
	  # add value and units if available
	  units=""
	  if [[ ${#datapt[@]} > 2 ]]; then
		units=${datapt[2]}
	  fi
	  printf " %d %s\n\n" $val "$units"
	  eval $color_rs 
  fi
done

SQL Output to Bars

With the base code I was able to start doing some more complicated actions, like piping SQL SELECT output. Below is an example from Sqlite3. The first step is to format the SQL output to: label,value;

pete@lubuntu:$ sqlite3 $HOME/dbs/someuser.db "select fname,age from users limit 4"
Brooke|18
Leah|18
Pete|100
Fred|77
pete@lubuntu:$ sqlite3 $HOME/dbs/someuser.db "select fname,age from users limit 4" \
>  | tr  '|' ',' | tr '\n' ';'
Brooke,18;Leah,18;Pete,100;Fred,77;
pete@lubuntu:$ sqlite3 $HOME/dbs/someuser.db "select fname,age from users limit 4" \ > | tr '|' ',' | tr '\n' ';' | ./hbars -t "Sqlite3 Users" -p Sqlite3 Users Brooke █████████ 18 Leah █████████ 18 Pete ██████████████████████████████████████████████████ 100 Fred ██████████████████████████████████████ 77

Web Page with Bars

Support was added for fixed chart width, engineering units and custom line colours. HTML <center> tags were used on the title.

$ cat data.txt
temp,44,degC,1;
humidity,33,%,4;
air pressure,88,mm Hg,5;
rain/precipation,6,mm,6;

$ cat data.txt | ./hbars -W -f 24 -w 50 -t '<center>Raspi Data</center>' > data.htm

Final Comments

Horizontal bars are relatives easy to create in Bash. Unfortunately showing vertical bars and Line charts will require a different approach

A Web Server in 1 Line of Bash Code

For people who do a lot of work with command line tools or Bash code, having a Bash web server could be very handy.

I was really amazed that in one line of Bash code I was able to create web servers that could:

  • send the output from a bash command directly to a browser page
  • create diagnostic pages using standard Linux tools
  • create pages that view Rasp PI GPIO pins
  • create a page to toggle a PI GPIO pin

One Line Web Servers

There are number of 1 line web servers, these are minimal offerings that exist in most of the programming languages.

A Bash web server can be created using the nc or netcat, the networking utility:

while true; do { \
  echo -ne "HTTP/1.0 200 OK\r\nContent-Length: $(wc -c <index.htm)\r\n\r\n"; \
  cat index.htm; } | nc -l -p 8080 ; \ 
done

This Bash statement echo’s to port 8080, the output is an HTTP header with the file content length defined. The cat command is used to show the HTML file.

This 1 line Bash example shows a single page (index.htm) which isn’t overly useful, there are other web server options that would work much better.

Where a Bash web server really stands out is in its ability to execute command line utilities or scripts and send the results to a web client.

Bash Web Server Calling Bash Commands

The output from command line utilities like iostat can be sent directly to a web client:

while true;
  do echo -e "HTTP/1.1 200 OK\n\n$(iostat)" \
  | nc -l -k -p 8080 -q 1; 
done

There are 2 important options that need to be set on nc, and they are: -k (this keeps the connection open after the first connection) and -q 1 (this closes the connection after 1 seconds, so another connection can occur). Depending on the complexity of the script that is used the -q timing may need to be adjusted.

The web page for the iostat command will look like:

Multiple Commands with Headings

Comments and multiply command line utilities can be defined as a variable that can be passed to the Bash Web server.

The figlet utility can be used to create custom ASCII headings, this is useful if you want to keep things simple without using HTML syntax. To install figlet in Ubuntu enter: sudo apt-get install figlet .

An example of using figlet headings and the sensors and vmstat utility is:

title1=$(figlet Sensors)
cmd1=$(sensors | sed -e 's/\°/ /g') # browser has problem with degrees, so remove
title2=$(figlet VMStat)
cmd2=$(vmstat)
thebody="$title1\n$cmd1\n$title2\n$cmd2"

while true;
  do echo -e "HTTP/1.1 200 OK\n\n$thebody" \
  | nc -l -p 8080 -q 1; 
done

Bash Web Server with Raspberry Pi GPIO

For many Raspberry Pi projects monitoring the status of the GPIO (General Purpose Input/Output) pins is quite important.

The Raspberry Pi gpio utility can be used to show the present status with the readall option:

pi@raspberrypi:~/pete $ gpio readall
 +-----+-----+---------+------+---+---Pi 3B--+---+------+---------+-----+-----+
 | BCM | wPi |   Name  | Mode | V | Physical | V | Mode | Name    | wPi | BCM |
 +-----+-----+---------+------+---+----++----+---+------+---------+-----+-----+
 |     |     |    3.3v |      |   |  1 || 2  |   |      | 5v      |     |     |
 |   2 |   8 |   SDA.1 | ALT0 | 1 |  3 || 4  |   |      | 5v      |     |     |
 |   3 |   9 |   SCL.1 | ALT0 | 1 |  5 || 6  |   |      | 0v      |     |     |
 |   4 |   7 | GPIO. 7 |  OUT | 0 |  7 || 8  | 0 | IN   | TxD     | 15  | 14  |
 |     |     |      0v |      |   |  9 || 10 | 1 | IN   | RxD     | 16  | 15  |
 |  17 |   0 | GPIO. 0 |   IN | 0 | 11 || 12 | 0 | IN   | GPIO. 1 | 1   | 18  |
 |  27 |   2 | GPIO. 2 |  OUT | 0 | 13 || 14 |   |      | 0v      |     |     |
 |  22 |   3 | GPIO. 3 |   IN | 0 | 15 || 16 | 0 | IN   | GPIO. 4 | 4   | 23  |
 |     |     |    3.3v |      |   | 17 || 18 | 0 | IN   | GPIO. 5 | 5   | 24  |
 |  10 |  12 |    MOSI | ALT0 | 0 | 19 || 20 |   |      | 0v      |     |     |
 |   9 |  13 |    MISO | ALT0 | 0 | 21 || 22 | 0 | IN   | GPIO. 6 | 6   | 25  |
 |  11 |  14 |    SCLK | ALT0 | 0 | 23 || 24 | 1 | OUT  | CE0     | 10  | 8   |
 |     |     |      0v |      |   | 25 || 26 | 1 | OUT  | CE1     | 11  | 7   |
 |   0 |  30 |   SDA.0 |   IN | 1 | 27 || 28 | 1 | IN   | SCL.0   | 31  | 1   |
 |   5 |  21 | GPIO.21 |  OUT | 0 | 29 || 30 |   |      | 0v      |     |     |
 |   6 |  22 | GPIO.22 |  OUT | 0 | 31 || 32 | 0 | OUT  | GPIO.26 | 26  | 12  |
 |  13 |  23 | GPIO.23 |  OUT | 0 | 33 || 34 |   |      | 0v      |     |     |
 |  19 |  24 | GPIO.24 |  OUT | 0 | 35 || 36 | 0 | IN   | GPIO.27 | 27  | 16  |
 |  26 |  25 | GPIO.25 |  OUT | 0 | 37 || 38 | 0 | OUT  | GPIO.28 | 28  | 20  |
 |     |     |      0v |      |   | 39 || 40 | 0 | OUT  | GPIO.29 | 29  | 21  |
 +-----+-----+---------+------+---+----++----+---+------+---------+-----+-----+
 | BCM | wPi |   Name  | Mode | V | Physical | V | Mode | Name    | wPi | BCM |
 +-----+-----+---------+------+---+---Pi 3B--+---+------+---------+-----+-----+

Rather than passing the Bash commands as a string, a small Bash script (web_body.sh) file can be created. This file will show the time and then call the gpio readall command.

#!/bin/bash
# web_body.sh - Show the time and PI GPIO pins

date $T

echo "$(gpio readall)"

The online Bash web server command is:

while true; do { \
  echo -ne "HTTP/1.1 200 OK\r\n"; sh web_body.sh; } \
  | nc -l -k -q 2 8080; \
done

The Web page for this script will look something like:

Send GPIO Writes from the Address Bar

Client side GET requests can be simulated on the browser address bar.

For example entering gpio write 7 1 on the address bar sents the Bash Server a GET request.

The HTTP request encodes spaces, so a space appears as a %20.

Bash code can be added to look for specific messages. In this case the “gpio write 7 1” or “gpio write 7 0” message can be search for, and if it is found then that exact message can be executed.

The Bash code can now be modified to ; look for the “GET gpio” message, then decode any HTTP %20 characters to spaces, next parse out the string to get the gpio message, and finally execute the required command. The code is below:

while true; 
  do { echo -ne "HTTP/1.1 200 OK\r\n"; sh web_body.sh; } | \
  nc -l -k -q 5 8080 | grep "GET /gpio" | \
  sed -e 's/%20/ /g' | \
  eval $( awk '{print substr($0,6,15) }') ;
done

With the new code, the gpio write is executed and the result can be seen in the web page.

Create an HTML Form

Entering commands on the command line works but it’s crude, a better way is to create an HTML Form.

The Bash web server code can remain exact the same as in the earlier example.

The original script (web_body.sh) file can be modified to made the output HTML format and three forms can be included. The first and second form will define the GET actions to turn on or off the GPIO pin and the third form will be used to refresh the page to check for GPIO changes.

#!/bin/bash
# web_body.sh - Show the time and PI GPIO pins
#             - Use HTML instead of text output
#             - Add forms for GPIO on/off, and a refresh
echo "
<!DOCTYPE html><html><head>
</head><body>
<h1>Bash Commands in a Web Page</h1>
<h2>Toggle Pin 7 On/Off</h2>
<form action='gpio write 7 0'>
 <input type='submit' value='OFF'> 
</form> 
<form action='gpio write 7 1'>
 <input type='submit' value='ON'>
</form>
<form action=''>
 <input type='submit' value='Refresh Page'>
</form>
<pre>
"

date $T

echo "$(gpio readall)"

echo "</pre></body></html>"

The client web page is now:

After turning on or off the GPIO pin, a refresh of the web page is required to see the new status.

Final Commands

A Bash Web Server is a quick and easy solution for viewing the output from Bash scripts and commands. I especially like the fact that I don’t need to install any special software and I don’t need to write any HTML code.

It is important to note that the number of concurrent connections is very low, (1/sec if the nc -q option is 1).

A Bash Web Server supports client side GET and POST requests, however for complex requirements the Bash code could start to get messy quickly and it would probably be best to look at another solution.

Simple TCP/UDP Bash Apps

There are some great communication protocols (MQTT, RabbitMQ, ReDis …) that are excellent for passing data between nodes.

For applications where you only need to do simple communications a couple of lines of Bash can be used with TCP or UDP sockets.

In this blog I wanted to document UDP/TCP communications using Linux Bash commands to:

  • define periodic 1-way communications
  • use progress bars to show data from remote nodes
  • remotely send commands to a Raspberry Pi
  • setup simple TCP backdoors

NC (NetCat) – for TCP and UDP Connections

In theory you should be able to create a input read via something like:

echo $(read < /dev/udp/127.0.0.1/9999)

Then do a write using:

echo "some text" > /dev/udp/127.0.0.1/9999)

Unfortunately Linux device connections are not fully reliable, especially on the read or listening side. However the write component appears to be fairly solid. Luckily there is a solid solution using the nc (NetCat) command line utility. The nc utility is typically preloaded on most Linux systems.

The nc utility supports both UDP (-u option) and TCP (default) connections.

To setup a UDP listener, use the IP address of the listener node, and select the -k option to allow multiple connections to occur:

nc -u -l -k 192.168.0.111  9999

For this example the listener’s IP is: 192.168.0.111, and port 9999 is used.

To do manual writes from the command line, enter:

 nc -u  192.168.0.111  9999

To send data from a script there are two methods, either using nc or writing to the device:

# writing via nc, -w0 send only 1 message 
echo "456" | nc -u -w0 192.168.0.111  9999
# writing via device:
echo "456" > /dev/udp/192.168.0.111/9999

Multiple Writes and Zenity Progress Dialogs

Zenity is command line dialog utility that is typically preinstalled on most versions of Linux.

The data that is sent to the UDP listener can be piped to Zenity progress bar:

nc -u -l -k 192.168.0.111  9999 | zenity --progress --title="Remote Data"

A script to send seconds every second would be:

#!/bin/bash
echo "Press [CTRL+C] to stop..." 
( 
while : 
do
# $(date +'%S') seconds" | nc -u -w0 192.168.0.111 9999
 echo "$(date +'%S')" | nc -u -w0 192.168.0.111 9999
 sleep 1 
done 
)

When the script echos an integer the progress bar will be updated with the integer value. An echo string starting with a “#” will update the text above the bar.

The progress bar is from 0-100%, but the integer value can be re-scaled to make the information clearer. For example to re-scale 0-60 secs to 0-100:

echo "$(date +'%S')*100/60" | bc | nc -u -w0 192.168.0.111 9999

YAD – for Multiple Progress Bars

YAD (Yet Another Dialog) is a command line GUI utility that offers a little more functionality than Zenity. To install YAD on Raspberry Pi’s and Ubuntu: sudo apt-get install yad

A bash command with a UDP listener with YAD 2-bars would be:

nc -u -l -k 192.168.0.111  9999 | yad --multi-progress \
  --bar="CPU Idle" --bar="CPU Temp" --title="Remote CPU Info"

The CPU Idle Time can be found by:

top -n 1 | grep %Cpu | awk '{print $8}'
93.8

The CPU Temperature on a PC can be found by:

sensors | grep CPU | awk '{print substr($2,2,4)}'
44.0

A script to send the CPU Idle Time and Temperature to the UDP listener is:

#!/bin/bash
echo "Press [CTRL+C] to stop..." 
( 
while : 
do
 cpuidle=$(top -n 1 | grep %Cpu | awk '{print $8}') 
 echo "1:"$cpuidle | nc -u -w0 192.168.0.111 9999

 cputemp=$(sensors | grep CPU | awk '{print substr($2,2,4)}')
 echo "2:"$cputemp | nc -u -w0 192.168.0.111 9999
 echo "2:#"$cputemp" Deg C" | nc -u -w0 192.168.0.111 9999
 sleep 5 
done 
)

For YAD multiple progress bars, an echo of 1: is for bar 1, 2: is for bar 2 etc. Echo-ing “2:# ” updates the text for the 2nd bar.

NC is not the same on Rasp Pi

I found that on the Rasp Pi the nc listening functions would not pass any information to bash scripts. Manual mode still works to view messages, but the messages can’t be piped to other commands.

This mean that things like the Zenity and YAD progress bars would not work on a Raspberry Pi. For many applications this may not be a big problem because the Rasp Pi can still send information via nc.

NC vs. NCAT

The ncat utility is very similar to nc but it offers the ability to run commands. By default nc is preloaded on most systems, but ncat needs to be installed. Installing ncat will vary based on your OS.

The ncat utility allows you to make backdoors so be careful of its use.

To create a backdoor simply (via TCP), define the ncat -c (execute command option) to be /bin/bash:

ncat -l -k 192.168.0.108  9999 -c /bin/bash

If on a remote node you enter: ncat 192.168.0.108 , you can start typing commands that are run on the remote node with the results echoing back. Very cool for test system but super dangerous for real systems.

Remotely Toggle a Rasp Pi GPIO Pin

Rather than opening up the system totally fixed commands can be defined. For example to toggle pin 7 on a Rasp Pi. A listener script is run:

ncat -l -k 192.168.0.108  9999 -c "gpio toggle 7"

A remote button GUI script could be used in conjunction with the listener script to toggle the GPIO pin:

#!/bin/bash
#
# Toggle a Rasp Pi GPIO pin

rc=1 # OK button return code =0 , all others =1
while [ $rc -eq 1 ]; do
  ans=$(zenity --info --title 'Remote Connect to Pi' \
      --text 'Toggle GPIO Pin' \
      --ok-label Quit \
      --extra-button TOGGLE \
       )
  rc=$?
  echo "${rc}-${ans}"
  echo $ans
  if [[ $ans = "TOGGLE" ]]
  then
        echo "Toggle Pin"
        nc -w0 192.168.0.108  9999
  fi
done

Send a Command String to Run Remotely

For this example the Rasp Pi is setup to be TCP listener, and the command (-c option) is /bin/bash, so this allows the remote PC to send custom commands:

ncat -l -k 192.168.0.108  9999 -c /bin/bash

On the remote system a bash script is created with 2 buttons and the custom commands are sent to the Pi to run:

#!/bin/bash
#
# Toggle two Rasp Pi GPIO pins

rc=1 # OK button return code =0 , all others =1
while [ $rc -eq 1 ]; do
  ans=$(zenity --info --title 'Remote Connect to Pi' \
      --text 'Toggle GPIO Pins' \
      --ok-label Quit \
      --extra-button Pin2 \
      --extra-button Pin7 \
       )
  rc=$?
  echo "${rc}-${ans}"
  echo $ans
  if [[ $ans = "Pin2" ]]
  then
        echo "gpio toggle 2" | nc -w0 192.168.0.108  9999
  elif [[ $ans = "Pin7" ]]
  then
        echo "gpio toggle 7" | nc -w0 192.168.0.108  9999
  fi
done

On systems other than Rasp Pi, the nc command can also be used to run remote programs by:

nc -u -l -k 192.168.0.111  9999 | awk '{ system($1 " " $2 " " $3 " " $4)}'

Final Comments

In this blog I’ve kept things focused on TCP/UDP communications with bash script but you could easily include Arduino, Python and Node-Red as either clients or servers.

Zenity: Command line Dialogs

Zenity is command line GUI creator that has been around since 2012, and it is pre-installed on most versions of Linux, including Raspberry PI’s. Zenity is also available for MacOS and Windows.

I came across it on a recent project and I wanted to document some of my code. My examples include:

  • CPU stats on a dialog – 1 line of Bash script
  • Show a web page in a dialog – 1 line
  • Create a 4 button PI Rover control – ~ 25 lines
  • Dynamic bar of CPU core temperature – 7 lines
  • Show CSV or SQL data in a list dialog – 1 line
  • Form to insert user data in an SQL database – ~7 lines

What is Zenity?

Zenity is a command line dialog creator. I found it pretty quick to pick up and it works well for simple Bash scripts. Zenity supports:

  • Basic forms
  • Calendar dialogs
  • Color selection dialogs
  • File Selection Dialogs
  • List Dialog
  • Message and Notification Dialog
  • Progress bars and Scales
  • Text Entry and Text Information Dialogs

Message Dialogs

Message dialogs can be created for errors, info, questions or warnings. The difference is the icon that shows up (and Ok/Cancel for the question dialog).

The Bash code to get the instantaneous CPU idle time would be:

pi@raspberrypi:~ $ # Run top once and look for the line with %Cpu
pi@raspberrypi:~ $ top -n 1 | grep Cpu
%Cpu(s):  7.4 us,  3.6 sy,  0.0 ni, 88.8 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st

pi@raspberrypi:~ $ # Get the 8th item from the grep

pi@raspberrypi:~ $ top -n 1 | grep %Cpu | awk '{print $8}'
88.8

The bash code to show the CPU idle time in a info dialog is:

zenity --info --text=$(top -n 1 | grep %Cpu | awk '{print $8}') --title="CPU Idle Time"

Message Dialogs with Custom Font

Text font and size can be modified in message dialogs, using the Pango Markup Language syntax. Pongo is similar to HTML. The <span></span> set of tags is used to encode font and color definitions, For example:

zenity --warning --text='<span font="32" foreground="red">HIGH Temperature</span>' --title="HDD Check"

Unfortunately only Message Dialog texts can changed, so text in dialogs like list, scale, and progress can’t not have their fonts changed.

Web Pages in a Text-Info Dialog

Text or HTML files can be passed to a text-info dialog:

zenity --text-info --title="Background Reading" --html --url="https://developer.gnome.org"

A checkbox can be added, and the user feedback can be read by the bash script:

#!/bin/sh
# show web page in a dialog with a next step action
#
theurl="https://developer.gnome.org"

zenity --text-info --title="Background Reading" --html --url=$theurl \
       --checkbox="I read it...and I'm good to go"
rc=$?
echo $rc
case $rc in
    0)
        echo "Start some next step"
	# next step
	;;
    1)
        echo "Stop installation!"
	;;
   -1)
        echo "An unexpected error has occurred."
	;;
esac

It is important to note that the text-info dialog should be used for simple web pages, there is no Javascript support and web links will launch the default web browser with requested page.

Refreshing Message Dialogs

Of all the Zenity dialogs, only the Progress dialog supports a method to update text on an open dialog. A workaround is to use the –timeout option to close the dialog and then redisplay it with the new data.

rc=5
while [[ $rc -eq 5 ]];
do 
  zenity --info --text=$(date +'%S' ) \
  --title="Seconds Timer Test" --timeout=5 --ok-label Quit $ zenity --info \
  --text=$(date +'%S' )   --title="Seconds Timer Test" 2>/dev/null

  rc=$?	
  echo $rc
done 

This “timeout and redraw” method is ugly because the window always gets positioned in the middle of the screen and this can be quite annoying. Unfortunately Zenity does not support any top/left positioning options.

The xdotool can be used find the zenity window id and then reposition it, but this would need to be done in another script. (It can’t be done in the same script because the zenity line doesn’t complete until either it times out or OK is pressed). The xdotool script would be:

pid=$(xdotool search -onlyvisible -name myzenitywindownname)
xdotool windowmove $pid 0 0

If you need dynamically updated text on a dialog I think that it would be best to use another tool, (Python, YAD etc.).

Info Dialog with Extra Buttons – Pi Rover Controls

It’s possible to add some extra buttons to an info dialog. Below is an example where a Raspberry Pi Rover is controlled with a zenity multi-button info dialog. (Note: the pins will vary with your setup.)


#!/bin/bash
#
# rover.sh - Rover Controls with Multiple Button Dialog
# Define GPIO pins for the motors motorL=7 motorR=11 rc=1 # OK button return code =0 , all others =1 while [ $rc -eq 1 ]; do ans=$(zenity --info --title 'Drive a Rover' \ --text 'Motor Action' \ --ok-label Quit \ --extra-button FORWARD \ --extra-button STOP \ --extra-button LEFT \ --extra-button RIGHT \ ) rc=$? echo "${rc}-${ans}" echo $ans if [[ $ans = "FORWARD" ]] then echo "Running the Rover" gpio -1 write $motorL 1 ; gpio -1 write $motorR 1; elif [[ $ans = "STOP" ]] then echo "Stopping the Rover" gpio -1 write $motorL 0 ; gpio -1 write $motorR 0; elif [[ $ans = "LEFT" ]] then echo "Rover turning Left" gpio -1 write $motorL 1 ; gpio -1 write $motorR 0; elif [[ $ans = "RIGHT" ]] then echo "Rover turning RIGHT" gpio -1 write $motorL 0 ; gpio -1 write $motorR 1; fi done

The script can be run by: bash rover.sh

Below is the dialog and the rover.

Progress Bars – Show Dynamic Values

A Zenity progress dialog can show dynamic updates with scripts that define steps using sleep statements. When the step outputs a value the process bar is updated. The text on the progress dialog is changed by outputting a text string starting with a # character.

A 3-step example would be:

(
echo "33"; echo "# 1/3 done" ; sleep 5; \
echo "66"; echo "# 2/3 done" ; sleep 5; \
echo "100";echo "# Finished"  \
) | zenity --progress --title="3 step test"

The progress dialog can use a bash for or while statement. The progress dialog can be passed both a new text label and a value. A text string starting with # is interpreted as the new text label. A number string is interpreted as the progress bar value.

Below is an example where a value is counted from 1 to 100:

( for i in `seq 1 100`; do echo $i; echo "# $i";  sleep 1; done ) | zenity --progress

The next thing I tested is a dialog that runs indefinitely (or until you hit “Control-C”). It’s import to note that the progress bar is from 0-100, so scaling your value may be required. An example of scaling a time from 0-60 to 0-100 would be:

echo "$(date +'%S')*100/60" | bc

Using the above code to show seconds in a dialog would be:

#!/bin/bash
# show_sec.sh - progress dialog to show seconds
echo "Press [CTRL+C] to stop..." 
( 
  while :; do 
  echo "# $(date +'%S')" 
  # Scale 0-60 to 0-100 
  echo "$(date +'%S')*100/60" | bc
  sleep 1 
  done 
  ) | zenity --progress  --title="Show Time in Seconds"

To run this script: bash show_sec.sh

A more useful dialog would be to show the CPU temperature:

#!/bin/sh 
# show_cpu_temp.sh - Progress Dialog to show CPU temperature
# 
echo
 "Press [CTRL+C] to stop..."
(
while :; do 
  echo "# $(sensors | grep CPU)" 
  sensors | grep CPU | awk '{print substr($2,2,4) }' 
  sleep 5 
done ) | zenity --progress --title="CPU Temperature"  

List Dialog – Show CSV/SQL Data

If you are working with a simple known data set then the List Dialog might be a good fit.

The List Dialog expects the data to be a sequential list, so a 2 column example of static data would be:

zenity --list \
  --title="2 Column Example" \ 
  --column="Month" --column="Sales" \
   Jan 100 Feb 95 Mar 77 Apr 110 May 111

Text and CSV files can also be used in Zenity lists. The first step is to convert the file into a single column of data. This can be done with the tr statement. For the example below the comma (,) is replaced with a newline (\n) character:

$ cat lang.txt
Brazil,Brasilia,Portuguese
England,London,English
France,Paris,French
Germany,Berlin,German

$ cat lang.txt | tr ',' '\n'
Brazil
Brasilia
Portuguese
England
London
English
France
Paris
French
Germany
Berlin
German

Now the sequential data can be passed into a Zenity list:

cat lang.txt | tr ',' '\n' | zenity --list \
  --title="Country Info" \
  --column="Country" --column="Capital" --column="Language"

Once you have some Zenity and Bash basics down you can do some fairly advanced operations. Below is a 1-line example that uses awk to parse out specific fields (1 and 3) and then the user selected output is echo-ed.

awk -F "\"*,\"*" '{print $1 "\n" $3}' pidata.csv  | \
  echo $(zenity --list --column="field1" --column="field3" --print-column=ALL)

Almost all SQL servers have a command line interface. The output from an SQL query can be formatted to a “zenity friendly” form. The interface will vary from database to database, but an example with Sqlite would be:

(sqlite3 someuser.db "select fname,lname,age,job from users" ) | tr '|' '\n' | zenity --list \
  --title="My Database" \
  --column="first name" --column="last name" --column=age --column=job

Form Dialog – Insert SQL Data

The Forms Dialog allows for date, text and password inputs, and the result are passed as string (| is the default separator). A form example with output would be:

$ row=$(zenity --forms --title="Create user" --text="Add new user" \
   --add-entry="First Name" \    
   --add-entry="Last Name" \    
   --add-entry="Age" \    
   --add-entry="Job") ; echo $row

field1|field2|field3|field4

The next step is to format the form data into an SQL statement. The SQL INSERT syntax is:

INSERT INTO table (field1,field2…) VALUE (value1,value2…)

For the example above, field1|field2|field3|field4 needs to formatted to the values. This manipulation can be done by the bash sed command with search and replace (s) option:

$ row="field1|field2|field3|field4"
$ echo "'$row'" | sed "s/|/','/g"
'field1','field2','field3','field4'

The bash script to present the zenity form and input the data is below. The if statement is used to ensure that the cancel button wasn’t pressed. More if statements would probably be required for some data validation.

# zen_sqlin.sh - create a form to add a new user into a SQLite3 database
row=$(zenity --forms --title="Create user" --text="Add new user" \ --add-entry="First Name" \ --add-entry="Last Name" \ --add-entry="Age" \ --add-entry="Job") if [[ -n $row ]] # Some data found then indata=$(echo "'$row'" | sed "s/|/','/g") cmd="sqlite3 someuser.db \"INSERT INTO users (Fname,Lname,Age,Job) VALUES ($indata)\"" eval $cmd echo "Added data: $indata" fi

The script is run by: bash zen_sqlin.sh

Zenity (GTk) Warning Messages

Depending on your system you might see some Zenity warning messages such as:

Gtk-Message: 15:30:52.461: GtkDialog mapped without a transient parent. This is discouraged.

I never saw this on my Raspberry Pi but I did see it on my lubuntu system. To make things cleaner the warning can be piped to the null device:

$ zenity --info --text=$(date +'%S' )   --title="Seconds Timer Test" 2>/dev/null

Some Final Comments

I really just touched the surface on what zenity can do. For more info see some of the tutorials.

For simple stuff zenity works fine. If you’re looking for a more complete command line GUI tool try YAD, for myself I’ll stick to Python.

As a side note, there is a Python library for zenity. If you’re feeling comfortable with the bash version of zenity and you only need to do simple dialogs then this might be a good fit.