WOOB – Web Outside Of Browsers

The Web Outside of Browsers Project allows users to access Internet data in a generic way without using a Web Browser.

Woob offers a common interface for accessing a variety of different Internet data sources. So rather than using a specific API to access to weather and then another to access a job boards, the Woob API/tools can be used for both of these data sources.

Woob can be used as a:

  • Python library,
  • A command line tool, or by
  • Bash scripting

In this blog I wanted to document my Linux notes on:

  • basic command line tool usage
  • Bash/Python examples for getting weather data
  • Bash/Python examples for playing Internet streaming music

Note: WOOB is in development so features may change. Also there is a large list of application modules, but at present most are European based.

Getting Started

To install woob:

$ pip install woob

Once woob is installed a list of features can be shown by entering woob:

$ woob
usage: woob [--version] <command> [<args>]

Use one of this commands:
   bands           display bands and suggestions
   bank            manage bank accounts
   bill            get/download documents and bills
   books           manage rented books
   bugtracker      manage bug tracking issues
   calendar        see upcoming events
   cinema          search movies and persons around cinema
   cli             call a method on backends
   config          manage backends or register new accounts
   contentedit     manage websites content
   dating          interact with dating websites
   debug           debug backends
   gallery         browse and download web image galleries
   gauge           display sensors and gauges values
   geolocip        geolocalize IP addresses
   housing         search for housing
   job             search for a job
   lyrics          search and display song lyrics
   money           import bank accounts into Microsoft Money
   msg             send and receive message threads
   parcel          manage your parcels
   paste           post and get pastes from pastebins
   pricecompare    compare products
   radio           search, show or listen to radio stations
   recipes         search and consult recipes
   repos           manage a woob repository
   rpg             manage RPG data
   shop            obtain details and status of e-commerce orders
   smtp            daemon to send and check messages
   subtitles       search and download subtitles
   torrent         search and download torrents
   translate       translate text from one language to another
   travel          search for train stations and departures
   video           search and play videos
   weather         display weather and forecasts

For more information about a command, use:
   $ man woob-<command>
or
   $ woob <command> --help

Each features has a backend (database) associated with it. For example to define the weather.com backend for the weather command:

$ woob weather
Warning: there is currently no configured backend for weather
Do you want to configure backends? (Y/n): y

Available modules:
1) [ ] ilmatieteenlaitos   Get forecasts from the Ilmatieteenlaitos.fi website
2) [ ] lameteoagricole   lameteoagricole website
3) [ ] meteofrance       Get forecasts from the MeteoFrance website
4) [ ] weather           Get forecasts from weather.com
a) --all--               install all backends
q) --stop--

Select a backend to create (q to stop): 4
Backend "weather" successfully added.

If a backend needs to be removed, for example meteofrance, use the backend remove command:

$ wood weather backend remove meteofrance

To check which backends are defined, use the backend list command:

$ woob weather backend list
Enabled: weather

$ woob radio backend list
Enabled: freeteknomusic, somafm

Using the Weather Command

Woob offers a number of weather modules. A good generic option is weather.com.

To get started at the command line, enter: woob weather

$ woob weather 
Welcome to weather v3.0

Copyright(C) 2010-2022 Romain Bignon
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Lesser General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

Type "help" to display available commands.

Loaded backends: weather

weather> help
Weather Commands:
    cities PATTERN
    current CITY_ID
    debug
    forecasts CITY_ID

Woob Commands:
    formatter [list | FORMATTER [COMMAND] | option OPTION_NAME [on | off]]
    select [FIELD_NAME]... | "$direct" | "$full"
    ls [-d] [-U] [PATH]
    cd [PATH]
    condition [EXPRESSION | off]
    count [NUMBER | off]
    backends [ACTION] [BACKEND_NAME]...
    quit
    logging [LEVEL]

Type "help <command>" for more info about a command.

The first step is to search for a city with the cities option. Once a city is found the id (1-20) is used to show the current weather and the future forecast.

The example below searches for Toronto. The first item (Toronto, Ontario, Canada) is selected (current 1) to show the present Toronto weather. Toronto’s future weather can be checked with the forecast command (forecast 1).

Bash Scripting for Weather

The command line utility can be used with Bash script, (For more info on the interface).

A scripting example to show the current and forecast weather for Toronto (the first item in the list):

$ # use the -n 1 option to get the first item
$ woob weather cities 'Toronto' -n 1
d8ccf908e3c4c748e232720575df7cdbca6e0f1b412bca8595d8a28d0c28e8bc@weather — Toronto, Ontario, Canada

$ # get the id (first string) and save to a variable
$ city_id=$(woob weather cities 'Toronto' -n 1 | awk '{print $1}')

$ # pass the city id to get the weather current
$ woob weather current $city_id 
2022-12-08: -1 °C - 1030.2hPa (Rising) - humidity 73% - feels like -4 °C/24 °F - Fair

An example is to get the current weather for a place and show it as a Linux system tray notification:

$ # Create a notification note with current weather information
$ my_city="Sauble Beach"
$ city_id=$(woob weather cities "$my_city" -n 1 | awk '{print $1}')
$ notify-send -t 1000000 "$my_city" "$(woob weather current $city_id )

To show the long term forecast information in a Zenity dialog:

$ # Create an Info Dialog with weather forecast information
$ my_city="Sauble Beach"
$ city_id=$(woob weather cities "$my_city" -n 1 | awk '{print $1}')
$ wf=$(woob weather forecast $city_id)
$ zenity --width=650 --info --text="$wf" --title="$my_city"

Python Woob Weather Example

A Python example to select the first city in a list and then show the forecast would be:

#!/usr/bin/python3
#
# wweather.py - Search for weather forecast for a place
#
from woob.core import Woob
from woob.capabilities.weather import CapWeather

w=Woob()
w.load_backends(CapWeather)

# Find the id for a location
for city in w.iter_city_search("Sauble Beach"):
    # stop at first hit    
    print(city.id, city.name)
    break

# Get the Forecast for that location
for w0 in w.iter_forecast(city.id):
    print("Date:", w0.date)
    print("Forecast:", w0.text)
    print("High:", w0.high)
    print("Low", w0.low, "\n")

For more information on the Weather calls.

Using the Radio Command

There are a number of radio modules to choose from. The first step is to add at least one module:

$ woob radio
Warning: there is currently no configured backend for radio
Do you want to configure backends? (Y/n): y

Available modules:
1) [ ] audioaddict       Internet radios powered by audioaddict.com services
2) [ ] bandcamp          Bandcamp music website
3) [ ] freeteknomusic    freeteknomusic website
4) [ ] ina               INA French TV video archives
5) [ ] nectarine         Nectarine Demoscene Radio
6) [ ] nova              Nova French radio
7) [ ] ouifm             OÜI FM French radio
8) [ ] radiofrance       Radios of Radio France: Inter, Info, Bleu, Culture, Musique, FIP, Le Mouv'
9) [ ] somafm            SomaFM web radio
10) [ ] virginradio       VirginRadio french radio
a) --all--               install all backends
q) --stop--

Select a backend to create (q to stop): 9
Backend "somafm" successfully added.

After one of more radio modules are added, you can search for radio stations that play a genre:

$ woob radio
Welcome to radio v3.0

Type "help" to display available commands.

Loaded backends: freeteknomusic, somafm

radio> search radio trance
id: thetrip@somafm
title: The Trip
description: Progressive house / trance. Tip top tunes.
current: B With U (Salinas' Dub Mix) - Tommyboy And Soultan Feat. Zara
streams: ['130Kbps/aac' ('http://somafm.com/thetrip130.pls'), 'fast/mp3' ('http://somafm.com/thetrip.pls'), '64Kbps/aacp' ('http://somafm.com/thetrip64.pls'), '32Kbps/aacp' ('http://somafm.com/thetrip32.pls')]

radio:/search> play 1

For this radio station search of “trance” only 1 station was found, so play 1 is used to listen to that station. (Note: if 3 were returned then you could listen to the third station by: play 3).

Bash Scripting for Internet Radio

There are some woob options that will make scripting a little easier. Some genres like “rock” or “ambient” will return too many hits, so the -n option can be set to limit the number of returned items. The -s option will return only selected fields.

$ # Show the id and description 
$ #  for the first 2 ambient station 
$ woob radio search radio "ambient" -s id,description -n 2
id: deepspaceone@somafm
description: Deep ambient electronic, experimental and space music. For inner and outer space exploration.

id: groovesalad@somafm
description: A nicely chilled plate of ambient/downtempo beats and grooves.

An example script (wlisten.sh) takes a music genre entered on the command line and then finds and plays the first related station.

#!/bin/bash
#
# wlisten.sh - find a Woob Radio station and play it
#
echo -e "\nWoob Internet Radio Player\n"

echo -n "Enter a type of Music to listen to: " 
read mtype
echo "Now playing : $mtype music ..."
woob radio play $(woob radio search radio "$mtype" -n 1 -s id | awk '{print $2}' )

To run this script (wlisten.sh) to play a reggae station:

$ bash wlisten.sh

Woob Internet Radio Player

Enter a type of Music to listen to: reggae
Now playing : reggae music ...

This example only uses the first returned station. By using a couple of Zenity list dialongs a Bash script can be created to present a list of preset genres, then the user can select a specific station.

#!/bin/bash
#
# wradio.sh - find a Woob Radio station and play it
#
mtype=$(zenity --title="Play Internet Radio" \
	--list --height=500 --width=300 --column="Music Type" \
        80s Ambient Dance House Jazz Pop Rock Reggae Top Trance) 
# if a music type is selected then get stations
echo "Music Type: $mtype"
if [[ -n "$mtype" ]] ; then
  stn=$(woob radio search radio $mtype -f radio_list  | zenity --list \
    --title="Select Radio Station" \
    --column=station --column=description --width=900 --height=300)
  # Get the station ID, its the 1st part of the station string
  stn_id="${stn%%—*}"
  # If the station string is not empty, play the station
  if [[ -n "$stn_id" ]]  ; then
    echo "Station Id: $stn_id"
    woob radio play $stn_id
  fi
fi

Python Woob Radio Example

To get a selection of radio stations for a genre:

from woob.core import Woob
from woob.capabilities.radio import CapRadio

w = Woob()
w.load_backends(CapRadio)

genre="rock"

for stns in w.iter_radios_search(genre):
        print(stns.id, stns.title, stns.description, "\n")

See the Woob tech api docs for more details on the radio API.

A Python Tkinter GUI example to select a genre and then play a radio station:

#!/usr/bin/python3
#
# wradio.py - Tkinter GUI for Woob Radio    
#
from tkinter import *
from tkinter import ttk
import subprocess
from woob.core import Woob
from woob.capabilities.radio import CapRadio

w = Woob()
w.load_backends(CapRadio)

# Fill the tree with genre stations
def show_stations():
    genre = cbo.get()
    tree.delete(*tree.get_children())
    # Insert Rows into tree based on found radio stations
    for stns in w.iter_radios_search(genre):
        tree.insert('','end',text=stns.id,values=(stns.id, stns.title, stns.description))    

# Play selected station
def tree_getitem(a):
    curItem = tree.focus()
    stn_info=tree.item(curItem) # station is dictionary variable
    thestn=stn_info['text'] # get the station id
    p=subprocess.Popen(["killall", "woob"])
    p=subprocess.Popen(["killall", "mpv"])
    if thestn != "":
        labelText.set("Playing Station: " + thestn)
        p=subprocess.Popen(["woob", "radio", "play", thestn])

def stop_music():
    labelText.set("")
    p=subprocess.Popen(["killall", "woob"])
    p=subprocess.Popen(["killall", "mpv"])
    

# Create an instance of tkinter frame
root = Tk()
genre="Rock"
root.title('Play Internet Radio Stations')
root.geometry('900x300')

# Create a listbox of station genres
label1 = ttk.Label(text = "Enter a genre: ").grid(row=0,column=0)
cbo = ttk.Combobox(root,height = 1)
genres= {"80s","Ambient", "Dance", "House","Jazz","Pop","Rock", "Reggae","Trance"}
cbo['value'] = [m for m in genres]
cbo.grid(row=0, column=1)

# Create a button to update station tree
bt_update = ttk.Button(root,text="Select Genre", command=show_stations)
bt_update.grid(row=0, column=2)

# Create a tree list with 3 columns
tree = ttk.Treeview(root, column=("Id","Station", "Desciption"), show='headings') 
tree.column("#1", width=100)
tree.heading("#1", text="ID")
tree.column("#2", width=200)
tree.heading("#2", text="Station")
tree.column("#3", width=600)
tree.heading("#3", text="Description")
tree.grid(row=1, column=0, columnspan=3)
tree.bind('<ButtonRelease-1>', tree_getitem)

# Create a label to show station being played
labelText = StringVar()
label2 = ttk.Label(textvariable=labelText).grid(row=2,column=0)

# Add a button to start and stop music
bt_stop = ttk.Button(root,text="Stop Music", command=stop_music)
bt_stop.grid(row=2,column=1)

root.mainloop()
    

Summary

The Woob project has a lot of potential, however because it is still in the development stage it can be a little challenging to use.

Due to the lack of examples, I found that I was able to get up and running faster with Bash scripts rather than Python. To build a Bash script I would first test what I wanted to do in the online tool, then I would mimic this in Bash.

Charts in 1 Line of Bash

There are some great charting packages out there, but for quick tests and playing around it’s nice to keep things simple. I wanted to do some basic Raspberry Pi charting and found that with the Gnuplot utility it took as little as 1 line of Bash to get the job done.

This blog shows some examples, probably the key points are:

  • Data can be piped to Gnuplot
  • Bash While-loops can be used to create real time/dynamic plots

Single Line Chart from a File

Gnuplot plot has a full scripting interface that allows users to create very complex presentations.

Gnuplot script can also be passed on the command line, with the options: -p (persist, spawn chart) and -e (execute command):

$ # Plot a file with 5 points, show file first
$ cat 5pnts.txt
2.7
3.5
4.1
3.9
5.6
$ # Plot the file as a line
$ gnuplot -p -e "plot '5pnts.txt' with lines title '5 Test Points' "

Multi-line Chart

The multi-line example is like the single line chart with the exception that the using statement is called for each plot line, 1:2 is the first line (x-axis column):(y-axis column) , and 1:3 is the second series line.

$ # Show the first 5 line of the data
$ head -n 5 data1.txt, Note: Gnuplot skips row starting with "#"
#t  Angle Error 
0.0	-14.7	3.6
1.0	8.6	    3.6
2.1	28.8	3.0
3.1	46.7	3.4
$
$ # plot, first column is x, next 2 columns are y series
$ gnuplot -p -e "plot 'data1.txt' \
  using 1:2 with lines title 'Angle','data1.txt' \
  using 1:3 with lines title 'Error'"

For-Loop Piped to a Line Chart

Rather than creating a text file, the output from a group of echo statements can be piped to Gnuplot. The ‘<cat’ definition is used for piped input.

$# Plot 10 random points
$ ( for i in `seq 1 10`; do echo "$RANDOM\n";  done )| \
  gnuplot -p -e "plot '<cat' with lines title '10 Random Pts'"

A more complex example would be to add a time on the X-axis and run a calculation at a sampled interval.

# Create a calc function
mycalc() {
  # Show the time and cpu idle time
  thetime=$(date +'%T')
  idle=$(top -n 1 | grep id | awk '{print $8}')
  echo "$thetime $idle\n"
}
# Run the calc function in the for-loop
# Define the y range, x scale as time
( for i in `seq 1 20`; do mycalc; sleep 1;  done )| \
  gnuplot -p -e "set yrange [50:100]; set xdata time; \
  set timefmt '%H:%M:%S';set format x '%H:%M:%S'; \
  plot '<cat' using 1:2 with lines title 'CPU Idle Time' "

Bar Charts from a File

For the bar chart example the plot options are defined in a variable. This example uses the iostat command to get the user and idle time usage. Awk is used to parse the 4th row and then get the individual values.

# Create a file with user and idle CPU usage
# Column: 1=bar position, 2=value, 3=label
iostat | awk '{if(NR==4) print "0 " $1 " user"}' > cpu0.dat
iostat | awk '{if(NR==4) print "1 " $3 " idle" }' >> cpu0.dat

# Define the options as a variable
options="set title 'CPU Diagnostics'; set boxwidth 0.5; \
  set style fill solid; "

# Plot the bars with labels, note: the label offset 
gnuplot -p -e "$options ;plot 'cpu0.dat' \
 using 1:2:xtic(3) with boxes notitle, \
 '' using 1:2:2 with labels font 'Helvetica,15' \
 offset 0.9,0.8  notitle" 

Bars with Dynamic Updates

A Bash while-loop can be used to dynamically get new data and pass it to Gnuplot.

Below is an example that refreshes a bar chart every 5 seconds. The refresh time is shown in the bar series title.

mycalc2() {
  # Show the time and user/cpu idle time
  # note: top gets instantaneous values, iostat uses averages
  top -n 1 | grep %Cpu | awk '{print "0 " $2 " user"}' > cpu0.dat
  top -n 1 | grep %Cpu | awk '{print "1 " $8 " idle"}' >> cpu0.dat
}
# define chart options
options="set title 'CPU Diagnostics'; set boxwidth 0.5; \
  set style fill solid; set yrange [0:100]; set xrange [-0.5:1.5]"

# create an infinite while loop to get data, and then plot
# note1: gnuplot needs a while loop to replot
# note2: use a big pause time in gnuplot or exiting will be tough 
( while :; do mycalc2; sleep 10;  done )| \
gnuplot -p -e "$options ; plot 'cpu0.dat' \
 using 1:2:xtic(3) with boxes \
 title \"$(date '+%T')\" ; while (1) { replot; pause 5 }" 

Summary

Using Gnuplot with the command line option allows for some quick and easy charting. For more detailed work try creating a Gnuplot script file.

Gemini Protocol for Lightweight Internet Apps

The Gemini protocol is a very simple and light Internet protocol.

Unlike HTML files that contain layers of tags, style sheets and Javascript, a Gemini document is a simple readable document.

In this blog I wanted to document:

  • Gemini servers and clients using only 1 line of Bash
  • How to use large ASCII text in Gemini documents
  • How to create simple bar charts
  • How to create Gemini CGI apps in Python

Getting Started

A Gemini document only supports a few statements and graphic images such as JPEG or PNG are not supported.

An example Gemini document with the common formatting options:

# Heading level 1 (H1) 
## Heading level 2 (H2)
### Heading level 3 (H3)

=> testpage.gmi A link to another page.

> This line will show as a block-quote. 

A list of items
* This is the first list item.
* This is another list item.

```
Code or ASCII Block 
   _   ___  ___ ___ ___   _____       _   
  /_\ / __|/ __|_ _|_ _| |_   _|__ __| |_ 
 / _ \\__ \ (__ | | | |    | |/ -_|_-<  _|
/_/ \_\___/\___|___|___|   |_|\___/__/\__|

```

Within a Gemini browser this file would look like:

Content Type

The content type is used by browsers and applications to determine how to manage the requested file.

Typically the content type is managed by server, for example a web server will send a HTTP/1.0 200 OK prior to sending the HTML file.

For the Gemini protocol the content type is: 20 text/gemini . Depending on the Gemini server the user may need to be add the content type manually. (More about this in Bash and CGI servers).

Simple Bash Gemini Servers and Clients

For basic testing a one line Bash statement can be used for custom Gemini servers and clients.

The Gemini protocol typically uses SSL (Secure Sockets Layer) and TLS (Transport Layer Security) encryption so the Bash ncat utility is needed (Note: the nc command does not support SSL).

Below is an example of single Gemini request:

The Gemini server is defined by using the -l , listen option. When a client requests data, the cat statement is used with a pipe (|) to output the file testpage.gmi.

The Gemini client echo’s a “from PC” message with its request, this helps identify which client is requesting data.

A simple Bash ncat statement doesn’t manage the content type so a “20 text/gemini” line is added to the top of the test page.

Dynamic Bash Data

In the earlier example the Bash server is only supporting 1 request then exiting.

A while loop can be added to pass a Bash script to the ncat statement. Below is an example of Bash script (showdata.sh) that show CPU data using the vmstat utility:

#!/bin/bash
#
# showdata.sh - Output data for Gemini Bash Server 
#
echo "20 text/gemini"
echo  " "
echo "#VMSTAT"
echo  " "
date +"%T"
echo  " "
# set Gemini formating for ASCII
echo  "\`\`\`"
#vmstat
top -n 1
echo  "\`\`\`"

To make the script executable use the command: chmod +x showdata.sh

The Bash command to run this script as a Gemini server is:

while true; do ./showdata.sh | ncat  -l -p 1965 --ssl; done

The earlier Bash Gemini client command can be used, or a Gemini browser/client app can be used. The handling of SSL/TLS encryption will vary with the client app. I used the very basic Zain app (https://gitgud.io/sathariel/zain) :

(Note: for the Zain client I needed to load tcl/tls, sudo apt-get install -y tcl-tls)

Using a 1 line Bash Gemini server is great for basic testing but I wouldn’t recommend if you want to connect to variety of different Gemini client applications.

Large ASCII Text

Gemini documents don’t support different font sizes, a workaround is to use the figlet tool to generate multi-line text strings. Figlet is installed on Ubuntu/Debian/Raspbian by:

sudo apt install figlet

Figlet has a number of font styles that use 2-5 line height characters:

When using ASCII headings the Gemini code formatting option should be used, and this has three backticks (“`) before and after the headings.

The earlier example can be modified to have ASCII headings:

#!/bin/bash
#
# showdata2.sh - Output Large Headings to a Gemini Bash Server 
#
echo "20 text/gemini"
echo  " "
echo  "\`\`\`"
# Generate large text 
figlet -f standard "Raspberry Pi"
figlet -f small -m 2  $(date +"%T")
# show CPU stats
vmstat
echo  "\`\`\`"

Bar Charts

Horizontal ASCII bar charts can be created by using the printf statement with different ASCII fill characters. For example:

# show a label with bar and value text 
#
label="temp"; val=20;
bar="printf '█%.0s' {1..$val}" ; 
printf '\n%-5s ' $label; eval $bar
printf '░%.0s' {1..5} ;
printf ' 50 C\n'

temp  ████████████████████░░░░░ 50 C

This bar logic can be using in a Raspberry Pi Stats page that looks at idle time and free space on the SD card:

#!/bin/bash
#
# pi_stats.sh - Output Pi Stats as a Gemini Page
#
echo "20 text/gemini"
echo  " "
# Put the rest of the Gemini document into code block mode
echo  "\`\`\`"
# Generate large text 
figlet -f standard "Pi Stats"

# Show the time
echo "Time: $(date +'%T')"
echo ""

# Get idle time, scale 0-50
idle=$(top -n 1 | grep id | awk '{print $8}')
barsize=$(echo $idle | awk '{printf "%0.f" , $1/2}')
thebar="printf '█%.0s' {1..$barsize}"
graysize=$(expr 50 - $barsize)
thegray="printf '░%.0s' {1..$graysize}"
printf 'Idle Time  '; eval $thebar; eval $thegray ; echo " $idle %"
echo  ""

# Get free space on SD card, scale 0-50
freesp=$(df | grep root | awk '{printf "%0.f", $5/2}')
barsize=$(echo $freesp | awk '{printf "%0.f" , $1/2}')
thebar="printf '█%.0s' {1..$barsize}"
graysize=$(expr 50 - $barsize)
thegray="printf '░%.0s' {1..$graysize}"
printf 'Free Space '; eval $thebar; eval $thegray ; echo " $freesp %"

echo  "\`\`\`"

To run this page use

while true; do ./pi_stats.sh | ncat  -l -p 1965 --ssl; done

Python CGI Pages

There are a number of good Gemini servers, for my testing I used the Python based Jetforce server, it is installed by:

pip install jetforce

To run the Jetforce server it is important to define a home directory, the allowable hosts that can connect (0.0.0.0 is all IP4 nodes) and the server’s hostname:

# Start the jetforce Gemini server for all IP4 hosts
jetforce --dir /home/pi/temp --host "0.0.0.0" --hostname 192.168.0.105 &
# Start jetforce without hardcoding hostname
# jetforce --dir /home/pi/temp --host "0.0.0.0" --hostname $(hostname -I) &

By default CGI (Common Gateway Interface) files are defined in the directory cgi-bin which is under the home directory.

Unlike a one-line Bash server, Jetforce server will pass environment variables like QUERY_STRING and host and remote connection information.

CGI programs can be written in a variety of programming languages. For this example I wanted to pass Raspberry Pi BME280 temperature, pressure and humidity sensor information to a Gemini CGI page.

The CGI program was written in Python and I installed a bme280 and figlet library:

pip install RPi.bme280
pip install pyfiglet

The Python script (sersor2gmi.py) outputs a Gemini content type, a Figlet title and then the sensor data values:

#!//usr/bin/python3
#
# sersor2gmi.py - send BME280 sensor data to a Gemini page
#
import smbus2
import bme280
import pyfiglet

# find device address via: i2cdetect -y 1
port = 1
address = 0x77
bus = smbus2.SMBus(port)

calibration_params = bme280.load_calibration_params(bus, address)

# the sample method returns a compensated_reading object
data = bme280.sample(bus, address, calibration_params)

# Output a Gemini page
print("20 text/gemini") #Note: Some Gemini CGI servers may do this
print("```") # use code mode
#print(pyfiglet.figlet_format("Pi BME280 Data", font = "small"))
print(pyfiglet.figlet_format("Pi BME 280 Data", font = "small"))

print("Temperature:" + "{:5.1f}".format(data.temperature) + " C" )
print("Pressure:   " + "{:5.1f}".format(data.pressure) + " kPa" )
print("Humidity:   " + "{:5.1f}".format(data.humidity) + " %" )
print("```") 

This file was added to the cgi-bin directory and it is made executeable (chmod +x sersor2gmi.py).

Below is the output seen in the Lagrange Gemini browser:

Final Comments

There are a variety of Gemini browsers for Windows, Android, Mac and Linux so if you’re looking for a quick and dirty internet solution Gemini might be a good solution.

I like how Gemini documents are totally readable, I can’t say the same for most web pages.

The one thing that I missed with Gemini pages is the ability of show nice charts, text based bars work for simple stuff but doing text based line charts is a little too ugly for me.

Memcached: fast lightweight network cache

For many projects an SQL database is overkill for simple storage of values. As an alternate solution there are a number of excellent distributed in-memory caching systems that can be used.

Two of the most popular in-memory caching systems are Redis and Memcached.

I’m big fan of Redis, and I’ve enjoyed doing projects with it. Redis offer some awesome speed with a small footprint, and it has many features that make it even superior to a messaging system like MQTT.

However if you’re looking for something 100% dead simple you should take a look a memcached, it has a super simple setup and with only a dozen commands so you’ll get up and running in no time.

There are API’s in all the common programming languages. It only takes 1 line of Bash to read or write to memchached.

Getting Started

Memached can be installed on all major OS’s. To install it on Ubuntu/Raspberry Pi:

sudo apt-get install memcached

If you’re using Docker there are some lightweight memcached images (89MB) that can be used. To run the memcached docker image the –net host option should be used:

$ sudo docker run -it --net host memcached

The -it (interactive) option could be useful if there are any errors kicked out.

See the man pages for a full description of memcached options.

Telnet can be used to enter manual enter commands (port 11211 is the default):

$ telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
version
VERSION 1.6.12

lru_crawler metadump all
key=mynum exp=-1 la=1642363105 cas=2 fetch=no cls=1 size=66
END
quit
Connection closed by foreign host.

For more information on the commands and Telnet.

Bash Interface to Memcached

There are lot of help guides for other Python, PHP, GO etc., but not a lot on how to use Bash.

The Bash nc command can be used to read and write to socket. The -q option will close the socket after 0 seconds (after the echo command is sent). The default port is 11211 but this can be changed along with adding some security options. The first example is to use the stats command:

$ echo "stats" | nc -q 0  192.168.0.120 11211 
STAT pid 1
STAT uptime 722
STAT time 1642359428
STAT version 1.6.12
...
END

$ # Find the number of current key items
$ echo "stats" | nc -q 0  192.168.0.120 11211 | grep curr_items
STAT curr_items 1

Set/Get a Key and Value

To set an in-memory key-value store, the syntax is:

set mykey <flags> <ttl> <size>
value

The flags option is typical set to 0. The ttl “time to live” is in seconds, a ttl of 0 is indefinite.The size of the value also needs to be define. (This is taken care of in the Python, PHP… libraries). It’s important to note that a newline (\r\n) is required before the value.

To set a key to a variable mynum with a value of 55 and a indefinite time to live:

$ # hard code the arguments in a set key command
$ # Note: echo -e is used to pass the \r\n for new lines
$ echo -e "set mynum 0 0 2 \r\n55\r" | nc -q 0  127.0.0.1 11211
STORED

A more flexible approach would be to pass variables:

$ # Set a new key/value pair, with no flags and indefinite timeout
$ ipp="192.168.0.120 11211"
$ thekey="mykey1"
$ thenum=55
$ numsize=${#thenum}
$ # send command 
$ echo -e "set $thekey 0 0 $numsize\r\n$thenum\r" | nc -q 0  $ipp
STORED

Get a key/value

The get key command returns 3 lines with the value being the 2nd line. Some awk code can be used to parse out just the value.

$ # Get a new key/value pair
$ ipp="192.168.0.120 11211"
$ thekey="mykey1"
$ 
$ echo "get $thekey" | nc -q 0  $ipp
VALUE mykey1 0 2
60
END
$ # Get just the 2nd line with the value
$ echo "get $thekey" | nc -q 0  $ipp | awk '{if (NR == 2) print $0}'
55
$ 
$ # Store the result in a variable
$ mykey1=$(echo "get $thekey" | nc -q 0  $ipp | awk '{if (NR == 2) print $0}')
$ echo "mykey1 = $mykey1"
mykey1 = 55

Increment/Decrement a Value

The inc / decr commands will increase or decrease a numeric stored key value by a defined amount:

$ #incr/decr a value's number
#
ipp="192.168.0.120 11211"
thekey="mykey1"

# Get starting value
echo "get mynum" | nc -q 0 $ipp | awk '{if (NR == 2) print $0}'
55

thediff=10; # increase the value (only positive values)
# incr and show the value
echo "incr $thekey $thediff" | nc -q 0  $ipp
65

thediff=5; # decrease the value (only positive values)
# incr and show the value
echo "decr $thekey $thediff" | nc -q 0 $ipp
60

Prepending and Appending

Memcached does not have queue or list functionality, if you need this take a look at Redis.

Below is an example script that creates a diagnostic log in a key/value. The append command adds msg text to the end of the overall string.

#!/usr/bin/bash
#
# diagmsg.sh - append msgs to a memcached variable string
#
ipp="192.168.0.120 11211"
msg="1:00 - Base Software Loaded\r"
size=${#msg}

echo -e "set mymsg 0 0 $size\r\n$msg\n\r" | nc -q 0  $ipp

# Create an array of diagnostics
diagmsgs=("2:00 - System Started\r" "2:15 - Getting Data\r" "2:30 - Backing up\r") 
# Append array to value text
for msg in "${diagmsgs[@]}"
do
  size=${#msg}
  #echo "Size: $size $msg";
  echo -e "append mymsg 0 0 $size\r\n$msg\n\r" | nc -q 0  $ipp;
done

# Show the result
echo -e "\nDiagnostic Results\n" 
echo "get mymsg" | nc -q 0 $ipp 

The results of this script would be:

$ bash diagmsg.sh
STORED
STORED
STORED
STORED

Diagnostic Results

VALUE mymsg 0 92
1:00 - Base Software Loaded
2:00 - System Started
2:15 - Getting Data
2:30 - Backing up

END

Final Thought

I played with the Python and PHP library and they were quite easy to use.

By adding a Bash interface to memcache it allows me to use programs like Octave/Matlib where a native interface isn’t available (just use the System call and pass the Bash code).

Sample Code: PHP

PHP example:

<?php
// memc.php - test memcached
//
// Load the PHP memcached component:
//
//    sudo apt-get install php-memcached

$memcache = new Memcached();
$memcache->addServer('localhost', 11211) or die ("Could not connect");

$version = $memcache->getVersion();
var_dump( $version);

$tmp_object = new stdClass;
$tmp_object->str_attr = 'test';
$tmp_object->int_attr = 123;

$memcache->set('key', $tmp_object);
echo "Store data in the cache (data will expire in 10 seconds)<br/>\n";

$get_result = $memcache->get('key');
echo "Data from the cache:<br/>\n";

var_dump($get_result);

?>

Sample Code: Python

Python example:

$ # Install Python memcache library
$ python3 -m pip install pymemcache

$ # Example set/get
$ python3
Python 3.8.10 (default, Nov 26 2021, 20:14:08) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pymemcache.client import base
>>> client = base.Client(('localhost', 11211))
>>> client.set('mykey','teststring')
True
>>> client.get('mykey')
b'teststring'
>>> client.set('mynum',44)
True
>>> client.get('mynum')
b'44'

Gnuplot Speedometer Gauge

Gnuplot is a great package that allows you to do charting from the command line.

Gnuplot has a wide range of plot types but unfortunately a speedometer gauge is not one of them.

This blog documents my notes in creating a dynamic gauge.

Some useful takeaways:

  • A gnuplot can be created with only graphic elements (no chart)
  • A named object can be re-positioned without redrawing the background
  • A gnuplot script can be called like a Bash or Python script with the first line being: #!/usr/bin/gnuplot

A Dynamic Gauge

The script gnuplot.gp is an example that refreshes a gauge chart every 5 seconds with the processor’s idle time. The idle time is obtained using the Linux top command (with some awk to get the 8th line item).

#!/usr/bin/gnuplot
#
# gnuplot.gp - speedometer dial with title as the description/value
#
set xrange [-1:1]
set yrange [0:1]
set angles degrees
set size ratio -1
# r1 = annulus outer radius, r2 = annulus inner radius
r1=1.0
r2=0.5
unset border; unset tics; unset key; unset raxis

set style fill solid noborder

# define a "needle" pointer as object 1
set object 1 circle at first 0,0 front size r1 arc [181:182]  fillcolor rgb 'black'

# define the gauge background
set object circle at first 0,0  size r1 arc [0:180]  fillcolor rgb 'green'
set object circle at first 0,0  size r1 arc [0:10]  fillcolor rgb 'red'
set object circle at first 0,0  size r1 arc [10:20]  fillcolor rgb 'yellow'
set object circle at first 0,0 front size r2 arc [0:180] fillcolor rgb 'black'

# plot the static background 
plot -10 notitle

# Define a partial heading  
variable = "Idle Time\n"
unit = "%"

# Refresh the plot with a new dial setting
while (1) {
  # Get the idle time using the top utility
  idle = system("top -n 1 | grep Cpu | awk '{print $8}'")
  # scale the value from 0-100 to 180-0 (Note: arc starts on the right)
  value = (100 - real(idle)) * 1.8
  # show the value in the title
  set title sprintf('%s %s %s', variable, idle, unit)  font "Ariel,28"
  # reposition the value in the gauge
  set object 1 circle at first 0,0 front size r1 arc [value:(value+2)]  fillcolor rgb 'black'
  replot
  pause 5 
}

The script can be run either from gnuplot or like a Bash script:

$ # run script from gnuplot
$ gnuplot -c gauge.gp
Use Control-C to exit ...
^C
$ # make script executable
$ chmod +x gauge.gp
$ # run script like a Bash script
$ ./gauge.gp
Use Control-C to exit ...
^C

Next Steps…

From here the next steps could be to add command line arguments to make the script more generic, such as:

  • pass the calculation to run (eg. pass the bash top command)
  • pass the scaling, title and units

Bash with MQTT

I’m working on re-purposing an old router. One of my goals is to bring sensor and performance data from the router to a Home Assistant node. The router doesn’t not have a lot of space so I’d prefer to use Bash rather than Python for MQTT communications.

While I was working on the project I wanted to use some simple tools to view the data, unfortunately I wasn’t able to find any information on how to make a good MQTT client in Bash.

This blog documents how I used Bash to show bar charts of MQTT data.

Setup

The first step is to install the Mosquitto command line tools on the OpenWrt router:

# Update the package manager list
opkg update
# Install the MQTT client utiliy
opkg install mosquitto-client-nossl

When I have some time I’d like to look at making some more MQTT Bash scripts that could be used with file input.

The next step is to install the Mosquitto client on my Linux PC and Raspberry Pi:

sudo apt-get install mosquitto-clients

There are a number of MQTT brokers that can be used, both Home Assistant and Node Red have reliable brokers. The Mosquitto broker can also be loaded on a Linux, MacOS and Windows node.

Publish MQTT Data in Bash

The Mosquitto client can be used to both publish and subscribe to MQTT data.

For my router project, I wanted the: temperature (from a USB thermometer), idle time, % used space and available space. Below is the script that passed the data to the mosquitto_pub (publishing) tool:

#!/usr/ash
#
# mqtt_data.sh - send data to MQTT broker
#
# Get Data values
temp=$(/usbstick/temper.sh)
idle=$(vmstat | awk '{ if (NR==3) print $15}')
used=$(df | awk '{if (NR==4) printf "%.f\n", $5 }')
space=$(df | awk '{if (NR==4) printf "%.1f\n", $4/1000 }')

echo "$temp $idle $used $space"

# Publish Data
server="192.168.0.111"
pause=2

mosquitto_pub -h $server -t rtr_temp -m $temp
sleep $pause
mosquitto_pub -h $server -t rtr_idle -m $idle
sleep $pause
mosquitto_pub -h $server -t rtr_used -m $used
sleep $pause
mosquitto_pub -h $server -t rtr_space -m $space

I added a pause between each publish to let me watch the actions.

The cron utility can be used to schedule the running of the script. Once a minute is the fastest time available with cron, so if faster times are needed the script could cycle with a while loop.

Read/Subscribe to MQTT Data

The mosquitto_sub client tool can be used to read or subscribe to the data.

To look at the router points:

$ service="192.168.0.111"
$ mosquitto_sub -h $server -v -t rtr_idle -t rtr_temp -t rtr_used -t rtr_space

rtr_temp 26.25
rtr_idle 100
rtr_used 66
rtr_space 1.1

The -v (–verbose) option show output with the topic names and the values. Topics, -t option, can be put on the command line or passing in as a file.

For single point monitoring the Zenity utility can be used. This utility is typically preloaded on most Linux and Rasp Pi systems. A script to create a progress bar dialog with MQTT data is:

# Send MQTT values to a progress bar
( 
  while :; do
  msg=$(mosquitto_sub -h 192.168.0.111 -C 1 -t rtr_temp) 
  echo $msg
  echo "#$msg  Deg C"
  done
  ) | zenity --progress  --title="Router External Temperature"

Multipoint Progress Dialogs

The Zenity utility can only manage single point progress bars. For multiple point progress bars the YAD (Yet Another Dialog) package can be used.

To install YAD on Raspberry Pi’s and Ubuntu: sudo apt-get install yad

Below is some script that will show multiple points in a YAD dialog:

#!/usr/bash
#
# mqtt_bars.sh - Show multiple MQTT Topics on a Dialog
#
server="192.168.0.111"
topics=("rtr_idle" "rtr_temp" "rtr_used" )
scale=(100 40 100 )
units=( '%' 'degC' '%'  )
title="Router MQTT Points"

#Build topic and yad strings
yadstr=" "
for i in ${topics[@]}; do
  topstr="$topstr -t $i"
  yadstr="$yadstr --bar=$i"
done

echo "Press [CTRL+C] to stop..."
# Cycle thru 1 message at a time to YAD 
(
while : 
do 
  msg=$(mosquitto_sub -h $server -v -C 1 $topstr) 
  IFS=' ' read -a data <<< "$msg" 
  # match returned msg to order of bars, write value/label
  for i in "${!topics[@]} "
  do 
    if [ "${data[0]}" = "${topics[i]}" ]
    then 
      let j=i+1 ; # YAD indices start at 1
      # Rescale bar to defined scale
      barsize=$(bc <<< "${data[1]}*100/${scale[i]}")
      #barsize=100
      echo "$j:$barsize"
      echo  "$j:#${data[1]} ${units[i]}"
  fi
  done
done 
)  | yad --multi-progress $yadstr --title $title

Final Comments

There are probably lots of good 3rd party tools like Gnuplot that could be connected to mosquitto_sub to show real charts.

Bash Bar Charts for Text and Web Pages

This blog documents my notes on creating Bash text bar charts. The key topics are:

  • Create a simple horizontal bars (3 lines)
  • Create dynamic bars (~25 lines for an array of data values)
  • Add support for piped and command line data
  • Include options for: help, resizing, titles
  • Add colour and HTML support

A Horizontal Bar

To create a static bar an printf statements can be used. The seq {0..10} can be used to repeat an ASCII █ fill character 10 times.

$ printf 'LABEL: ' ; \
  printf '█%.0s' {1..10} ; \
  printf ' 10\n'

LABEL: ████████████████████ 10

Unfortunately the printf statement has some limitations on variable substitution. A simple workaround is to create a string and then eval it:

$ label="temp"; val=20;
$ bar="printf '█%.0s' {1..$val}" ; 
$ printf '\n%-5s ' $label; eval $bar ; printf ' %d\n' $val

temp  ████████████████████ 20

Coloured Bars

The tput setaf command can change the foreground, and tput setab is used for background colours. Colour codes are:

tput setab [1-7] # Set the background colour using ANSI escape
tput setaf [1-7] # Set the foreground colour using ANSI escape

Num  Colour    #define         R G B

0    black     COLOR_BLACK     0,0,0
1    red       COLOR_RED       1,0,0
2    green     COLOR_GREEN     0,1,0
3    yellow    COLOR_YELLOW    1,1,0
4    blue      COLOR_BLUE      0,0,1
5    magenta   COLOR_MAGENTA   1,0,1
6    cyan      COLOR_CYAN      0,1,1
7    white     COLOR_WHITE     1,1,1

To reset colours back to the defaults use: tput sgr0

An example to print a red bar and a stack of bars:

$ printf '\nLABEL: ' ; \
   tput setaf 1 ;\
   printf '█%.0s' {1..10} ; \
   printf ' 10\n'

LABEL: ██████████ 10
$ printf '\n 3 Stacked Bars: ' ; \
   tput setaf 1 ;\
   printf '█%.0s' {1..10} ; \
   tput setaf 2 ;\
   printf '█%.0s' {1..8} ; \
   tput setaf 4 ;\
   printf '█%.0s' {1..3} ; \
   printf ' 10+8+3=21\n'

 3 Stacked Bars: █████████████████████ 10+8+3=21

Dynamic Bars

The next step is to create a script that dynamically updates the bars. The tput clear command will clear the terminal screen keep the data and bars in the same location. The script below will dynamically show the CPU temperature, idle time and 2 random values with a 10 second update time.

#!/bin/bash
# 
# cpu_bars.sh - Show new data every 10 seconds
#
while :; do
    # Get data values
    CPUtemp=$(sensors | grep CPU | awk '{print substr($2,2,4)}')
    CPUidle=$(iostat | awk '{if (NR==4) print $6}')
    Random1=$((1+ $RANDOM % 100))
    Random2=$((1+ $RANDOM % 100))

    labels=( CPUtemp CPUidle Random1 Random2)
    values=( $CPUtemp $CPUidle $Random1 $Random2)
    units=( degC % psi mm)

    # Show a title
    tput clear
    printf " %10s " "" 
    tput setaf 7; tput smul;
    printf "%s\n\n" "Show CPU Data ($(date +%T'))"
    tput rmul;

    # cycle thru data and show a label, 
    for index in "${!labels[@]}"
    do
          tput setaf $(expr $index + 1); # don't use 0 (black) 
          printf " %10s " "${labels[index]}"
          eval "printf '█%.0s' {1..${values[index]%.*}}"
          printf " %s %s\n\n" ${values[index]} ${units[index]}
    done
    sleep 10
done

This script is run by: bash cpu_bars.sh .Typical output is below.

Bars in HTML

The ANSI colours are not supported in HTML, so instead HTML tags and style properties can be used.

It is important to use <pre> tags for Bash text output. Code to create two static bars in HTML would be:

$ (printf "<h1>A Bar from Bash</h1>\n" 
 printf "<pre><span style='font-size:24px;color:red'}>\n"
 printf 'LABEL1: ' ; printf '█%.0s' {1..10} ; printf ' 10\n'
 printf "</pre></span>\n") > bar1.htm

$ cat bar1.htm
<h1>A Bar from Bash</h1>
<pre><span style='font-size:24px;color:red'}>
LABEL1: ██████████ 10
</pre></span>

The script cpu_webbars.sh creates HTML output for an array of calculated values:

#!/bin/bash
# 
# cpu_webbars.sh - Show bars in HTML
#

# Get data values
CPUtemp=$(sensors | grep CPU | awk '{print substr($2,2,4)}')
CPUidle=$(iostat | awk '{if (NR==4) print $6}')
Random1=$((1+ $RANDOM % 100))
Random2=$((1+ $RANDOM % 100))

labels=( CPUtemp CPUidle Random1 Random2)
values=( $CPUtemp $CPUidle $Random1 $Random2)
units=( degC % psi mm)
colors=(red blue green magenta)

# Show a title
printf "<h1><center>Show CPU Data ($(date '+%T'))</center></h1>\n"
# cycle thru data and show a label, 
for index in "${!labels[@]}"
  do
  printf "<pre><span style='font-size:18px;color: ${colors[index]} '}>\n"
  printf " %10s " "${labels[index]}"
  eval "printf '█%.0s' {1..${values[index]%.*}}"
  printf " %s %s\n\n" ${values[index]} ${units[index]}
  printf "</pre></span>\n"
done

This script can be run and outputted to an file: bash cpu_webbars.sh > test.htm

Once the basic HTML output is working its possible to add headers and footers to make a more complete page:

header.htm > test.htm ; \
cat cpu_webbars.sh >> test.htm ; \
cat footer >> test.htm 

Piping Data to Bars

A script (hbar0.sh) will read the piped data and then create an array (data) of labels and values. The data array is cycled through and labels and bars are shown with a randomized colour:

#!/bin/bash
# hbar0.sh - Read in piped data  and plot bars
#     format: label,value;label2,value2;  and plot bars
#
input=$(< /dev/stdin) ; # read piped data
# remove new lines in files, and check for ";" after data pair 
input=$(echo $input | tr -d '\n')
IFS=';' read -r -a data <<< $input
printf "\n" 
for element in "${data[@]}"pete@lubuntu:~/Writing/Blog/text_bars
do 
  # make at array of each data element
  IFS=',' read -r -a datapt <<< $element
  # add a random color
  tput setaf $((1+ $RANDOM % 7))
  # print the label, bar and value
  printf " %10s " "${datapt##*[0]}"
  bar="printf '█%.0s' {1..${datapt[1]}}"
  eval $bar
  printf " %s\n\n" ${datapt[1]} 
  tput rmso ; # exit color mode   
done

The script can be tested with piped data:

$ echo "temp,33;pressure,44" | bash hbar0.sh

 temp       █████████████████████████████████ 33 

 pressure   ████████████████████████████████████████████ 44 

A data file can also be passed in using the cat command:

$ cat data0.txt 
temp,44;
humidity,33;
pressure,15;
wind spd,33;
wave ht,3;
$ cat data0.txt | bash hbar0.sh

       temp ████████████████████████████████████████████ 44

   humidity █████████████████████████████████ 33

   pressure ███████████████ 15

   wind spd █████████████████████████████████ 33

    wave ht ███ 3

Removing ANSI Colors

Terminal applications use ANSI color codes which unfortunately is not support on Web pages.

ANSI color codes can be removed from files and strings by:

# Strip out ANSI color codes:
cat infile | sed 's/\x1b\[[0-9;]*m//g' > outfile
$ echo "temp,33;pressure,44" | bash hbar0.sh > hbar0.txt
$ cat hbar0.txt

       temp █████████████████████████████████ 33

   pressure ████████████████████████████████████████████ 44

$ cat hbar0.txt | sed 's/\x1b\[[0-9;]*m//g'

       temp █████████████████████████████████ 33

   pressure ████████████████████████████████████████████ 44

A Final App

With the basics in place I was able to create an app that would support scaling, titles, units, custom colour and web output:

$ ./hbars
usage: hbars [data] [option] 
  -h --help     print this usage and exit
  -c --color    set color to all bars (default 7=white)
                 (0-grey,1-red,2=green,3=yellow,4=blue,5=magenta,6=cyan,7=white)
  -p --pretty   add different colors to bars (-c overrides)
  -t --title    top title
  -w --width    max width of bar (default 50)
  -W --Web      make output HTML formatted
  -f --fontsize fontsize for web output (default 24)

 examples:
   echo 'temp,33,C;pressure,14,psi' | ./hbars -t Weather -p -w 40 
   ./hbars -t Weather -p -w 40  'temp,33;pressure,14' 
   cat data.txt | ./hbars -W -f 24 -t 'Raspi Data' > data.htm

The code:

#!/bin/bash
#
# hbars.sh - show some text bars
#   pass data as:  label1,value1,unit1,color1; label2,value2,unit2,colour2; ....  
#
width=50
title=""
color=7
pretty=False
web=False
font=24

usage() { 
  echo "usage: hbars [data] [option] "
  echo "  -h --help     print this usage and exit"
  echo "  -c --color    set color to all bars (default 7=white)"
  echo "                 (0-grey,1-red,2=green,3=yellow,4=blue,5=magenta,6=cyan,7=white)"
  echo "  -p --pretty   add different colors to bars (-c overrides)"
  echo "  -t --title    top title"
  echo "  -w --width    max width of bar (default 50)"
  echo "  -W --Web      make output HTML formatted"
  echo "  -f --fontsize fontsize for web output (default 24)"
  echo ""
  echo " examples:"
  echo "   echo 'temp,33,C;pressure,14,psi' | ./hbars -t Weather -p -w 40 "
  echo "   ./hbars -t Weather -p -w 40  'temp,33;pressure,14' "
  echo "   cat data.txt | ./hbars -W -f 24 -t 'Raspi Data' > data.htm"
  echo ""

  exit 0
}
# Show help usage if no pipe data and no cmd line data
if [ -t 0 ]  && [ $# -eq 0 ] ; then
  usage
fi
# Check for command line options
while getopts "hpc:t:w:Wf:" arg; do
  case "$arg" in
    h) usage ;;
    c)  color=$OPTARG ;;
    p)  pretty=True; icolor=0 ;;
    t)  title=$OPTARG ;;
    w)  width=$OPTARG ;;
    W)  web=True;;
    f)  font=$OPTARG ;;
  esac
done
#------------------------------------------------
# Setup formatting for text, Web and color
# -----------------------------------------------
if [[ ${color} != 7 && ${pretty} = True ]]; then
  pretty=False
fi
colidx=0

setcolors() {
if [ $web = True ]; then
  colors=(gray red green yellow blue magenta cyan white)
  titlebold="echo '<h1>'"
  titlereset="echo '</h1>'"
  #color_set='echo "<span style=font-size:$(font)px  >" ' 
  #color_set="printf '<span  style=\"font-size:$(font)px;color:${colors[colidx]}\" >'" 
  color_set="printf '<pre><span style=\"color:${colors[colidx]} ; font-size:${font}px \" >'" 
  color_rs="echo '</span></pre>'"
else
  colors=(0 1 2 3 4 5 6 7 )
  titlebold="tput bold; tput smul"
  titlereset="tput rmul; tput rmso"
  color_set="tput setaf ${colors[colidx]}"
  color_rs="tput rmso"
fi
}
setcolors
#----------------------------------------------
# Get data, check if stdlin, file, if not assume string
#----------------------------------------------
if [ -p /dev/stdin ]; then
        lastarg=$(< /dev/stdin)
else
	lastarg=$(echo "${@: -1}")
	if test -f "$lastarg"; then
	  lastarg=$(<$lastarg)
	fi
fi
# Cleanup the input data
lastarg=$(echo $lastarg | sed 's/\n/;/g; s/  / /g; s/\t//g; s/;;/;/g')
IFS=';' read -r -a array <<< $lastarg

# ensure that there is some data
if [[ ${array} == 0 ]]; then
  echo "No data found"
  exit 0
fi
echo "input:$lastarg"
#exit 0
#------------------------------------
# Get max value and max label length
#------------------------------------
maxval=0
maxlbl=10
#echo "array:${array[@]}"
for element in "${array[@]}"
do 
  IFS=',' read -r -a datapt <<< $element
  if (( $(echo "$maxval < ${datapt[1]}" |bc -l) )); then
	maxval=${datapt[1]}
  fi
  if (( $(echo "$maxlbl < ${#datapt[0]}" |bc -l) )); then
	maxlbl=${#datapt[0]}
  fi
done
#---------------------------------
# Print Title - use bold/underline
#---------------------------------
if [[ ! -z $title ]]; then
  printf "\n %${maxlbl}s " " "
  eval $titlebold
  printf "%s" "${title}" ; printf "\n\n"
  eval $titlereset
fi
#------------------------------------
# Cycle thru data and build bar chart
#------------------------------------
for element in "${array[@]}"
do
# check data values
  IFS=',' read -r -a datapt <<< $element
  # check for empty records
  if [ ${#datapt[0]} = 0 ]; then 
    break
  fi
  label=${datapt[0]}
  if [[ ${label} != "-t*" ]]; then 
	  val=${datapt[1]}
	  sval=$(bc <<< "$width * $val / $maxval")

	  # add color, use 4th item if available
	  if [[ ${#datapt[@]} > 3 && $pretty = False ]]; then
		icolor=${datapt[3]}
	  fi
	  if [[ $pretty = True ]] ; then
		let colidx++
		if [ $colidx -gt 7 ]; then
		  let colidx=1
		fi
	  elif [[ ${#datapt[@]} > 3 ]]; then
		colidx=${datapt[3]}
	  else
	  	colidx=$color
	  fi
	  setcolors
          eval $color_set
          printf " %${maxlbl}s " "$label"
	  bar="printf '█%.0s' {1..$sval}"
	  eval $bar; 
	  # add value and units if available
	  units=""
	  if [[ ${#datapt[@]} > 2 ]]; then
		units=${datapt[2]}
	  fi
	  printf " %d %s\n\n" $val "$units"
	  eval $color_rs 
  fi
done

SQL Output to Bars

With the base code I was able to start doing some more complicated actions, like piping SQL SELECT output. Below is an example from Sqlite3. The first step is to format the SQL output to: label,value;

pete@lubuntu:$ sqlite3 $HOME/dbs/someuser.db "select fname,age from users limit 4"
Brooke|18
Leah|18
Pete|100
Fred|77
pete@lubuntu:$ sqlite3 $HOME/dbs/someuser.db "select fname,age from users limit 4" \
>  | tr  '|' ',' | tr '\n' ';'
Brooke,18;Leah,18;Pete,100;Fred,77;
pete@lubuntu:$ sqlite3 $HOME/dbs/someuser.db "select fname,age from users limit 4" \ > | tr '|' ',' | tr '\n' ';' | ./hbars -t "Sqlite3 Users" -p Sqlite3 Users Brooke █████████ 18 Leah █████████ 18 Pete ██████████████████████████████████████████████████ 100 Fred ██████████████████████████████████████ 77

Web Page with Bars

Support was added for fixed chart width, engineering units and custom line colours. HTML <center> tags were used on the title.

$ cat data.txt
temp,44,degC,1;
humidity,33,%,4;
air pressure,88,mm Hg,5;
rain/precipation,6,mm,6;

$ cat data.txt | ./hbars -W -f 24 -w 50 -t '<center>Raspi Data</center>' > data.htm

Final Comments

Horizontal bars are relatives easy to create in Bash. Unfortunately showing vertical bars and Line charts will require a different approach

A Web Server in 1 Line of Bash Code

For people who do a lot of work with command line tools or Bash code, having a Bash web server could be very handy.

I was really amazed that in one line of Bash code I was able to create web servers that could:

  • send the output from a bash command directly to a browser page
  • create diagnostic pages using standard Linux tools
  • create pages that view Rasp PI GPIO pins
  • create a page to toggle a PI GPIO pin

One Line Web Servers

There are number of 1 line web servers, these are minimal offerings that exist in most of the programming languages.

A Bash web server can be created using the nc or netcat, the networking utility:

while true; do { \
  echo -ne "HTTP/1.0 200 OK\r\nContent-Length: $(wc -c <index.htm)\r\n\r\n"; \
  cat index.htm; } | nc -l -p 8080 ; \ 
done

This Bash statement echo’s to port 8080, the output is an HTTP header with the file content length defined. The cat command is used to show the HTML file.

This 1 line Bash example shows a single page (index.htm) which isn’t overly useful, there are other web server options that would work much better.

Where a Bash web server really stands out is in its ability to execute command line utilities or scripts and send the results to a web client.

Bash Web Server Calling Bash Commands

The output from command line utilities like iostat can be sent directly to a web client:

while true;
  do echo -e "HTTP/1.1 200 OK\n\n$(iostat)" \
  | nc -l -k -p 8080 -q 1; 
done

There are 2 important options that need to be set on nc, and they are: -k (this keeps the connection open after the first connection) and -q 1 (this closes the connection after 1 seconds, so another connection can occur). Depending on the complexity of the script that is used the -q timing may need to be adjusted.

The web page for the iostat command will look like:

Multiple Commands with Headings

Comments and multiply command line utilities can be defined as a variable that can be passed to the Bash Web server.

The figlet utility can be used to create custom ASCII headings, this is useful if you want to keep things simple without using HTML syntax. To install figlet in Ubuntu enter: sudo apt-get install figlet .

An example of using figlet headings and the sensors and vmstat utility is:

title1=$(figlet Sensors)
cmd1=$(sensors | sed -e 's/\°/ /g') # browser has problem with degrees, so remove
title2=$(figlet VMStat)
cmd2=$(vmstat)
thebody="$title1\n$cmd1\n$title2\n$cmd2"

while true;
  do echo -e "HTTP/1.1 200 OK\n\n$thebody" \
  | nc -l -p 8080 -q 1; 
done

Bash Web Server with Raspberry Pi GPIO

For many Raspberry Pi projects monitoring the status of the GPIO (General Purpose Input/Output) pins is quite important.

The Raspberry Pi gpio utility can be used to show the present status with the readall option:

pi@raspberrypi:~/pete $ gpio readall
 +-----+-----+---------+------+---+---Pi 3B--+---+------+---------+-----+-----+
 | BCM | wPi |   Name  | Mode | V | Physical | V | Mode | Name    | wPi | BCM |
 +-----+-----+---------+------+---+----++----+---+------+---------+-----+-----+
 |     |     |    3.3v |      |   |  1 || 2  |   |      | 5v      |     |     |
 |   2 |   8 |   SDA.1 | ALT0 | 1 |  3 || 4  |   |      | 5v      |     |     |
 |   3 |   9 |   SCL.1 | ALT0 | 1 |  5 || 6  |   |      | 0v      |     |     |
 |   4 |   7 | GPIO. 7 |  OUT | 0 |  7 || 8  | 0 | IN   | TxD     | 15  | 14  |
 |     |     |      0v |      |   |  9 || 10 | 1 | IN   | RxD     | 16  | 15  |
 |  17 |   0 | GPIO. 0 |   IN | 0 | 11 || 12 | 0 | IN   | GPIO. 1 | 1   | 18  |
 |  27 |   2 | GPIO. 2 |  OUT | 0 | 13 || 14 |   |      | 0v      |     |     |
 |  22 |   3 | GPIO. 3 |   IN | 0 | 15 || 16 | 0 | IN   | GPIO. 4 | 4   | 23  |
 |     |     |    3.3v |      |   | 17 || 18 | 0 | IN   | GPIO. 5 | 5   | 24  |
 |  10 |  12 |    MOSI | ALT0 | 0 | 19 || 20 |   |      | 0v      |     |     |
 |   9 |  13 |    MISO | ALT0 | 0 | 21 || 22 | 0 | IN   | GPIO. 6 | 6   | 25  |
 |  11 |  14 |    SCLK | ALT0 | 0 | 23 || 24 | 1 | OUT  | CE0     | 10  | 8   |
 |     |     |      0v |      |   | 25 || 26 | 1 | OUT  | CE1     | 11  | 7   |
 |   0 |  30 |   SDA.0 |   IN | 1 | 27 || 28 | 1 | IN   | SCL.0   | 31  | 1   |
 |   5 |  21 | GPIO.21 |  OUT | 0 | 29 || 30 |   |      | 0v      |     |     |
 |   6 |  22 | GPIO.22 |  OUT | 0 | 31 || 32 | 0 | OUT  | GPIO.26 | 26  | 12  |
 |  13 |  23 | GPIO.23 |  OUT | 0 | 33 || 34 |   |      | 0v      |     |     |
 |  19 |  24 | GPIO.24 |  OUT | 0 | 35 || 36 | 0 | IN   | GPIO.27 | 27  | 16  |
 |  26 |  25 | GPIO.25 |  OUT | 0 | 37 || 38 | 0 | OUT  | GPIO.28 | 28  | 20  |
 |     |     |      0v |      |   | 39 || 40 | 0 | OUT  | GPIO.29 | 29  | 21  |
 +-----+-----+---------+------+---+----++----+---+------+---------+-----+-----+
 | BCM | wPi |   Name  | Mode | V | Physical | V | Mode | Name    | wPi | BCM |
 +-----+-----+---------+------+---+---Pi 3B--+---+------+---------+-----+-----+

Rather than passing the Bash commands as a string, a small Bash script (web_body.sh) file can be created. This file will show the time and then call the gpio readall command.

#!/bin/bash
# web_body.sh - Show the time and PI GPIO pins

date $T

echo "$(gpio readall)"

The online Bash web server command is:

while true; do { \
  echo -ne "HTTP/1.1 200 OK\r\n"; sh web_body.sh; } \
  | nc -l -k -q 2 8080; \
done

The Web page for this script will look something like:

Send GPIO Writes from the Address Bar

Client side GET requests can be simulated on the browser address bar.

For example entering gpio write 7 1 on the address bar sents the Bash Server a GET request.

The HTTP request encodes spaces, so a space appears as a %20.

Bash code can be added to look for specific messages. In this case the “gpio write 7 1” or “gpio write 7 0” message can be search for, and if it is found then that exact message can be executed.

The Bash code can now be modified to ; look for the “GET gpio” message, then decode any HTTP %20 characters to spaces, next parse out the string to get the gpio message, and finally execute the required command. The code is below:

while true; 
  do { echo -ne "HTTP/1.1 200 OK\r\n"; sh web_body.sh; } | \
  nc -l -k -q 5 8080 | grep "GET /gpio" | \
  sed -e 's/%20/ /g' | \
  eval $( awk '{print substr($0,6,15) }') ;
done

With the new code, the gpio write is executed and the result can be seen in the web page.

Create an HTML Form

Entering commands on the command line works but it’s crude, a better way is to create an HTML Form.

The Bash web server code can remain exact the same as in the earlier example.

The original script (web_body.sh) file can be modified to made the output HTML format and three forms can be included. The first and second form will define the GET actions to turn on or off the GPIO pin and the third form will be used to refresh the page to check for GPIO changes.

#!/bin/bash
# web_body.sh - Show the time and PI GPIO pins
#             - Use HTML instead of text output
#             - Add forms for GPIO on/off, and a refresh
echo "
<!DOCTYPE html><html><head>
</head><body>
<h1>Bash Commands in a Web Page</h1>
<h2>Toggle Pin 7 On/Off</h2>
<form action='gpio write 7 0'>
 <input type='submit' value='OFF'> 
</form> 
<form action='gpio write 7 1'>
 <input type='submit' value='ON'>
</form>
<form action=''>
 <input type='submit' value='Refresh Page'>
</form>
<pre>
"

date $T

echo "$(gpio readall)"

echo "</pre></body></html>"

The client web page is now:

After turning on or off the GPIO pin, a refresh of the web page is required to see the new status.

Final Commands

A Bash Web Server is a quick and easy solution for viewing the output from Bash scripts and commands. I especially like the fact that I don’t need to install any special software and I don’t need to write any HTML code.

It is important to note that the number of concurrent connections is very low, (1/sec if the nc -q option is 1).

A Bash Web Server supports client side GET and POST requests, however for complex requirements the Bash code could start to get messy quickly and it would probably be best to look at another solution.

Simple TCP/UDP Bash Apps

There are some great communication protocols (MQTT, RabbitMQ, ReDis …) that are excellent for passing data between nodes.

For applications where you only need to do simple communications a couple of lines of Bash can be used with TCP or UDP sockets.

In this blog I wanted to document UDP/TCP communications using Linux Bash commands to:

  • define periodic 1-way communications
  • use progress bars to show data from remote nodes
  • remotely send commands to a Raspberry Pi
  • setup simple TCP backdoors

NC (NetCat) – for TCP and UDP Connections

In theory you should be able to create a input read via something like:

echo $(read < /dev/udp/127.0.0.1/9999)

Then do a write using:

echo "some text" > /dev/udp/127.0.0.1/9999)

Unfortunately Linux device connections are not fully reliable, especially on the read or listening side. However the write component appears to be fairly solid. Luckily there is a solid solution using the nc (NetCat) command line utility. The nc utility is typically preloaded on most Linux systems.

The nc utility supports both UDP (-u option) and TCP (default) connections.

To setup a UDP listener, use the IP address of the listener node, and select the -k option to allow multiple connections to occur:

nc -u -l -k 192.168.0.111  9999

For this example the listener’s IP is: 192.168.0.111, and port 9999 is used.

To do manual writes from the command line, enter:

 nc -u  192.168.0.111  9999

To send data from a script there are two methods, either using nc or writing to the device:

# writing via nc, -w0 send only 1 message 
echo "456" | nc -u -w0 192.168.0.111  9999
# writing via device:
echo "456" > /dev/udp/192.168.0.111/9999

Multiple Writes and Zenity Progress Dialogs

Zenity is command line dialog utility that is typically preinstalled on most versions of Linux.

The data that is sent to the UDP listener can be piped to Zenity progress bar:

nc -u -l -k 192.168.0.111  9999 | zenity --progress --title="Remote Data"

A script to send seconds every second would be:

#!/bin/bash
echo "Press [CTRL+C] to stop..." 
( 
while : 
do
# $(date +'%S') seconds" | nc -u -w0 192.168.0.111 9999
 echo "$(date +'%S')" | nc -u -w0 192.168.0.111 9999
 sleep 1 
done 
)

When the script echos an integer the progress bar will be updated with the integer value. An echo string starting with a “#” will update the text above the bar.

The progress bar is from 0-100%, but the integer value can be re-scaled to make the information clearer. For example to re-scale 0-60 secs to 0-100:

echo "$(date +'%S')*100/60" | bc | nc -u -w0 192.168.0.111 9999

YAD – for Multiple Progress Bars

YAD (Yet Another Dialog) is a command line GUI utility that offers a little more functionality than Zenity. To install YAD on Raspberry Pi’s and Ubuntu: sudo apt-get install yad

A bash command with a UDP listener with YAD 2-bars would be:

nc -u -l -k 192.168.0.111  9999 | yad --multi-progress \
  --bar="CPU Idle" --bar="CPU Temp" --title="Remote CPU Info"

The CPU Idle Time can be found by:

top -n 1 | grep %Cpu | awk '{print $8}'
93.8

The CPU Temperature on a PC can be found by:

sensors | grep CPU | awk '{print substr($2,2,4)}'
44.0

A script to send the CPU Idle Time and Temperature to the UDP listener is:

#!/bin/bash
echo "Press [CTRL+C] to stop..." 
( 
while : 
do
 cpuidle=$(top -n 1 | grep %Cpu | awk '{print $8}') 
 echo "1:"$cpuidle | nc -u -w0 192.168.0.111 9999

 cputemp=$(sensors | grep CPU | awk '{print substr($2,2,4)}')
 echo "2:"$cputemp | nc -u -w0 192.168.0.111 9999
 echo "2:#"$cputemp" Deg C" | nc -u -w0 192.168.0.111 9999
 sleep 5 
done 
)

For YAD multiple progress bars, an echo of 1: is for bar 1, 2: is for bar 2 etc. Echo-ing “2:# ” updates the text for the 2nd bar.

NC is not the same on Rasp Pi

I found that on the Rasp Pi the nc listening functions would not pass any information to bash scripts. Manual mode still works to view messages, but the messages can’t be piped to other commands.

This mean that things like the Zenity and YAD progress bars would not work on a Raspberry Pi. For many applications this may not be a big problem because the Rasp Pi can still send information via nc.

NC vs. NCAT

The ncat utility is very similar to nc but it offers the ability to run commands. By default nc is preloaded on most systems, but ncat needs to be installed. Installing ncat will vary based on your OS.

The ncat utility allows you to make backdoors so be careful of its use.

To create a backdoor simply (via TCP), define the ncat -c (execute command option) to be /bin/bash:

ncat -l -k 192.168.0.108  9999 -c /bin/bash

If on a remote node you enter: ncat 192.168.0.108 , you can start typing commands that are run on the remote node with the results echoing back. Very cool for test system but super dangerous for real systems.

Remotely Toggle a Rasp Pi GPIO Pin

Rather than opening up the system totally fixed commands can be defined. For example to toggle pin 7 on a Rasp Pi. A listener script is run:

ncat -l -k 192.168.0.108  9999 -c "gpio toggle 7"

A remote button GUI script could be used in conjunction with the listener script to toggle the GPIO pin:

#!/bin/bash
#
# Toggle a Rasp Pi GPIO pin

rc=1 # OK button return code =0 , all others =1
while [ $rc -eq 1 ]; do
  ans=$(zenity --info --title 'Remote Connect to Pi' \
      --text 'Toggle GPIO Pin' \
      --ok-label Quit \
      --extra-button TOGGLE \
       )
  rc=$?
  echo "${rc}-${ans}"
  echo $ans
  if [[ $ans = "TOGGLE" ]]
  then
        echo "Toggle Pin"
        nc -w0 192.168.0.108  9999
  fi
done

Send a Command String to Run Remotely

For this example the Rasp Pi is setup to be TCP listener, and the command (-c option) is /bin/bash, so this allows the remote PC to send custom commands:

ncat -l -k 192.168.0.108  9999 -c /bin/bash

On the remote system a bash script is created with 2 buttons and the custom commands are sent to the Pi to run:

#!/bin/bash
#
# Toggle two Rasp Pi GPIO pins

rc=1 # OK button return code =0 , all others =1
while [ $rc -eq 1 ]; do
  ans=$(zenity --info --title 'Remote Connect to Pi' \
      --text 'Toggle GPIO Pins' \
      --ok-label Quit \
      --extra-button Pin2 \
      --extra-button Pin7 \
       )
  rc=$?
  echo "${rc}-${ans}"
  echo $ans
  if [[ $ans = "Pin2" ]]
  then
        echo "gpio toggle 2" | nc -w0 192.168.0.108  9999
  elif [[ $ans = "Pin7" ]]
  then
        echo "gpio toggle 7" | nc -w0 192.168.0.108  9999
  fi
done

On systems other than Rasp Pi, the nc command can also be used to run remote programs by:

nc -u -l -k 192.168.0.111  9999 | awk '{ system($1 " " $2 " " $3 " " $4)}'

Final Comments

In this blog I’ve kept things focused on TCP/UDP communications with bash script but you could easily include Arduino, Python and Node-Red as either clients or servers.