1-Line of Bash to add Apps to the System Tray

Putting your commonly used apps and scripts into the Linux system tray could be quite useful.

In this blog I’ll look at two approaches:

  • alltray – a command line utility to dock any program into the system tray
  • yad – (Yet Another Dialog) tool is a Bash GUI builder that also supports trays notifications.

The alltray approach is dead simple and it works for any linux application or script. The yad utility offers a little more functionality by adding the ability to create command line dialogs and it can dynamically change the tray icon, actions, menus and tool tip.

Alltray

To install alltray in Debian/Raspian/Ubuntu:

sudo apt install alltray

An simple alltray example that calls an xterm window with the top utility (to show top running processes):

# Use alltray to put a terminal window app in the the tray
#  usage: alltray [options] ["] <program_name> [parameters] ["]
#
alltray  "xterm -hold -T 'Top Processes' -e 'top'"

As a default alltray will use the icon of the command that is being called.

For this example I used xterm to open a new terminal with the options of: -hold (keep teminal open) , -t ( add title) and -e (execute a command).

If you want to change the font name and size, use the -fa and -fs options, for example:

xterm -hold -fa Monospace -fs 14  -T "Top Processes" -e  "top"

Alltray also supports a custom icon and right-click menus, an example of this would be:

# Show a tray item with a custom icon and right-click menu options
#   syntax for adding menus is: --menu "menu-label: command"
#
alltray  "xterm -hold -T 'Top Processes' -e  'top'" \
  -i /usr/share/icons/Adwaita/256x256/legacy/face-glasses.png \
  --menu "Disk Usage:xterm -hold -T 'df' -e 'df'" \
  --menu "Sensors:xterm -hold -T 'Sensors' -e 'sensors' " 

For this example a custom icon (face-glasses.png) is added along with 2 right-click menu options.

YAD – Yet Another Dialog

The yad utility is a command line dialog tool that supports a good selection of different dialog types, also yad can be configured for system tray applications without needing alltray.

For Debian/Raspian and Ubuntu systems yad can be installed by:

sudo apt install yad

Yad has a lot of options, (see the man pages or help for more details). To create a simple two button dialog:

# Show a simple YAD dialog
yad  --text="SOME TEXT"  --title="My Dialog"

To put this simple yad dialog on the system tray with a custom icon:

# Create a YAD system tray item to call a YALL dialog
#
yad --notification --image="gtk-execute" \ 
  --command="yad --text='SOME TEXT' --title='My Dialog' " \
  --text="My Tooltip"

Icons are fairly easy to manage using the yad tool: yad-icon-browser .

Like alltray, yad supports menus, below is a menu example:

# Create a YAD system tray item with a right-click menu
#
yad --notification --image="gtk-execute" \
 --command="yad --text='SOME TEXT' --title='My Dialog'" \
 --menu="Memory! yad --text='$(vmstat)' --title=VMSTAT \
        | Sensors! yad --text='$(sensors)' --title=Sensors \
        | USB ! yad --text='$(lsusb)' --title=USB \
        | Quit ! killall yad" \
 --text="My Tooltip"

The syntax for menus is:

menu=STRING
              STRING must be in the form:
              menu_label1[! action1[! icon1]]|label2[! action2[! icon2]]....   
              Menus are separated with `|' or --separator  argument.
              Menu items are separated with `!' or --item-separator argument.

For this menuing example I passed the output from command line tools like vmstat, sensors and lsusb to the yad –text parameter.

Unlike alltray, yad doesn’t have a built in quit menu option, but this functionality can be added with:

Quit ! killall yad

Remotely Change a YAD Tray Item

The yad notification option has a –listen parameter that allows commands to be sent from stdin (standard input) to yad in the form command:args. Possible commands are icon, tooltip, visible, action, menu and quit.

The yad stdio can be redirected a named pipe and this will enable other bash scripts to be able to send it commands. Below is a basic Bash script that creates a named pipe variable (mytraypipe=”/tmp/tray1.pipe”) and then it creates the named pipe if it doesn’t exist.

The Bash command: exec 1<> $mytraypipe , redirects stdio (file 1) to the named pipe. The final step is to call the yad with <&1 , to redirect the stdio into the command.

#!/bin/bash
#
# dyn_tray.sh - create a system tray item that can be modified
#             - write changes to the named pipe: $mytraypipe 
#
mytraypipe="/tmp/tray1.pipe"

# Make the pipe if required
if ! test -e "$mytraypipe"; then
  mkfifo $mytraypipe
fi

# redirect the stdio (file 1) to the named pipe
exec 1<> $mytraypipe

# create the notification icon
yad --notification                  \
    --listen                        \
    --image="emblem-colors-grey"              \
    --text="Dummy tooltip"   \
    --command="yad --text='Test Tray App' " <&1

Below is an example that changes the icon and tool tip for the earlier Bash script. This first step is to define a variable with the correct named pipe, after this commands can be send with an echo statement to the named pipe.

Summary

If you are looking for a quick way to pull together some commonly used apps and scripts both alltray and yad offer a simple 1 line of Bash solution.

If you want to dynamically change the tray item then yad is the tool for you.

Charts in 1 Line of Bash

There are some great charting packages out there, but for quick tests and playing around it’s nice to keep things simple. I wanted to do some basic Raspberry Pi charting and found that with the Gnuplot utility it took as little as 1 line of Bash to get the job done.

This blog shows some examples, probably the key points are:

  • Data can be piped to Gnuplot
  • Bash While-loops can be used to create real time/dynamic plots

Single Line Chart from a File

Gnuplot plot has a full scripting interface that allows users to create very complex presentations.

Gnuplot script can also be passed on the command line, with the options: -p (persist, spawn chart) and -e (execute command):

$ # Plot a file with 5 points, show file first
$ cat 5pnts.txt
2.7
3.5
4.1
3.9
5.6
$ # Plot the file as a line
$ gnuplot -p -e "plot '5pnts.txt' with lines title '5 Test Points' "

Multi-line Chart

The multi-line example is like the single line chart with the exception that the using statement is called for each plot line, 1:2 is the first line (x-axis column):(y-axis column) , and 1:3 is the second series line.

$ # Show the first 5 line of the data
$ head -n 5 data1.txt, Note: Gnuplot skips row starting with "#"
#t  Angle Error 
0.0	-14.7	3.6
1.0	8.6	    3.6
2.1	28.8	3.0
3.1	46.7	3.4
$
$ # plot, first column is x, next 2 columns are y series
$ gnuplot -p -e "plot 'data1.txt' \
  using 1:2 with lines title 'Angle','data1.txt' \
  using 1:3 with lines title 'Error'"

For-Loop Piped to a Line Chart

Rather than creating a text file, the output from a group of echo statements can be piped to Gnuplot. The ‘<cat’ definition is used for piped input.

$# Plot 10 random points
$ ( for i in `seq 1 10`; do echo "$RANDOM\n";  done )| \
  gnuplot -p -e "plot '<cat' with lines title '10 Random Pts'"

A more complex example would be to add a time on the X-axis and run a calculation at a sampled interval.

# Create a calc function
mycalc() {
  # Show the time and cpu idle time
  thetime=$(date +'%T')
  idle=$(top -n 1 | grep id | awk '{print $8}')
  echo "$thetime $idle\n"
}
# Run the calc function in the for-loop
# Define the y range, x scale as time
( for i in `seq 1 20`; do mycalc; sleep 1;  done )| \
  gnuplot -p -e "set yrange [50:100]; set xdata time; \
  set timefmt '%H:%M:%S';set format x '%H:%M:%S'; \
  plot '<cat' using 1:2 with lines title 'CPU Idle Time' "

Bar Charts from a File

For the bar chart example the plot options are defined in a variable. This example uses the iostat command to get the user and idle time usage. Awk is used to parse the 4th row and then get the individual values.

# Create a file with user and idle CPU usage
# Column: 1=bar position, 2=value, 3=label
iostat | awk '{if(NR==4) print "0 " $1 " user"}' > cpu0.dat
iostat | awk '{if(NR==4) print "1 " $3 " idle" }' >> cpu0.dat

# Define the options as a variable
options="set title 'CPU Diagnostics'; set boxwidth 0.5; \
  set style fill solid; "

# Plot the bars with labels, note: the label offset 
gnuplot -p -e "$options ;plot 'cpu0.dat' \
 using 1:2:xtic(3) with boxes notitle, \
 '' using 1:2:2 with labels font 'Helvetica,15' \
 offset 0.9,0.8  notitle" 

Bars with Dynamic Updates

A Bash while-loop can be used to dynamically get new data and pass it to Gnuplot.

Below is an example that refreshes a bar chart every 5 seconds. The refresh time is shown in the bar series title.

mycalc2() {
  # Show the time and user/cpu idle time
  # note: top gets instantaneous values, iostat uses averages
  top -n 1 | grep %Cpu | awk '{print "0 " $2 " user"}' > cpu0.dat
  top -n 1 | grep %Cpu | awk '{print "1 " $8 " idle"}' >> cpu0.dat
}
# define chart options
options="set title 'CPU Diagnostics'; set boxwidth 0.5; \
  set style fill solid; set yrange [0:100]; set xrange [-0.5:1.5]"

# create an infinite while loop to get data, and then plot
# note1: gnuplot needs a while loop to replot
# note2: use a big pause time in gnuplot or exiting will be tough 
( while :; do mycalc2; sleep 10;  done )| \
gnuplot -p -e "$options ; plot 'cpu0.dat' \
 using 1:2:xtic(3) with boxes \
 title \"$(date '+%T')\" ; while (1) { replot; pause 5 }" 

Summary

Using Gnuplot with the command line option allows for some quick and easy charting. For more detailed work try creating a Gnuplot script file.

Web Scraping with 1 Line of Bash

In the past Python with the Beautiful Soup library has been a great approach for my web scraping.

I was recently doing a small project and I was amazing at what one Bash statement would get me.

My approach was to use the text based Lynx browser and pipe the output to a grep search.

Below is an example where I used Lynx to dump the “Sunshine Village Snow Forecast” web page to find how much snow they had.

The Lynx Text Browser

The first step in web scraping is to get a web page into a searchable format.

I started out by looking at using cURL with the html2text tool, but I found that using the Lynx browser offered a one step solution with a cleaner text output.

To install Lynx on Raspian/Debian/Ubuntu use:

sudo apt install lynx

The lynx -dump option will output a web page to text with HTML tags and Javascript removed. It’s important to note that what you see on the page may not match the outputted text.

Below is an example where I wanted to get the new snow at Sunshine Village. On the web page Javascript is used to show the snow depth as either centimetres or inches, but on the text output both units and their values are shown.

Bash has a good selection of string manipulation tools. Below is an example to extract the first part of string to only show the snow in centimeters (cm):

$ theurl="https://www.snow-forecast.com/resorts/Sunshine/6day/mid"
$ thestr="New snow in Sunshine Village:"
$ # Create a variable with the result string from Lynx
$ newsnow=$(lynx -dump "$theurl" | grep "$thestr")
$ 
$ echo "$newsnow"
   New snow in Sunshine Village:  4.8cm1.9in on Fri 8th (after 3 PM)
$    
$ # Get the first part of the string before "cm"
$ # The %% gets the first part
$ echo "${newsnow%%cm*} cm"
   New snow in Sunshine Village:  4.8 cm

My Final App

We were going on a family ski trip and to get pumped I created a morning notification script that showed the new morning snow and the base.

#!/bin/bash
#
# skitrip.sh - show the Sunshine ski conditions in a notification
#
theurl="https://www.snow-forecast.com/resorts/Sunshine/6day/mid"

# Get the new snow depth
thestr="New snow in Sunshine Village:"
result=$(lynx -dump "$theurl" | grep "$thestr")
newsnow="${result%%cm*} cm"

# Get the base
thestr="Top Lift:"
base=$(lynx -dump "$theurl" | grep "$thestr")

# Show the results in a desktop notification
msg="$newsnow\n$base (base)"
icon="$HOME/Downloads/mountain.png"
notify-send -t 10000000 -i "$icon"  "Sunshine Ski Resort" "$msg"

The notify-send utility will put a message on a Linux desktop, another option could be to send a SMS message.

Summary

Scraping web pages can be tricky and the pages can change at anytime.

I found that Lynx worked on many pages but not all.

The grep utility is extremely useful and it offers a lot of interesting options, such as getting lines before or after the found string.

Text Interfaces with Whiptail and Dialog

Text interfaces are extremely useful when an MS-Window client is trying to connect into a Linux or Raspberry Pi node.

There are some good options for creating Bash text interfaces, two of the most popular are Whiptail and Dialog. Raspberry Pi users will be familiar with Pi configuration tool, raspi-config, which uses Whiptail to create all its menu and application screens.

Whiptail comes pre-installed on Raspbian and on many Linux images. Whiptail is a lighter version of the more complete Dialog package.

This blog will show some examples using Whiptail and Dialog.

Getting Started

For my work I preferred using Dialog, I found that it had many more options and a slightly easier interface than Whiptail. However for projects where I needed to distribute my code to other users or nodes I would keep things simple and use Whiptail.

To install Whiptail and Dialog on a Raspbian/Debian/Ubuntu:

sudo apt install whiptail
sudo apt install dialog

A simple Whiptail test to show the date/time in a message box:

whiptail --title "Time/Date" --msgbox  "$(date)" 0 0

This statement will clear the terminal screen and show a default background with message box centered in the window. A height and width of : 0 0 will autosize the message box.

Refreshing YESNO Dialog

A message box only has a single OK button. A yesno dialog has two button.

Whiptail only supports a basic yesno function. The Dialog utility supports changing the button labels and a timeout so the window can be automatically refreshed.

Below is an example that refreshed an yesno dialog every 5 seconds. The show_dlg function generates a random temperature and humidity valid, and then calls the dialog utility. The YES button, relabelled as “Refresh”, will manually force a refresh of the data and redraw the window. The NO button, relabelled as “Cancel” will close the script and clear the screen.

#!/usr/bin/bash
#
# dyesno.sh - A freshing yes/no dialog with simulated sensor data

YES=0
TIMEOUT=255

show_dlg() {  
   # simulate some data
   data1="Temperature: $(( ( RANDOM % 10 )  + 20 )) C";   
   data2="Humidity: $(( ( RANDOM % 10 )  + 30 )) %";   
   message="$data1\n\n$data2";   
   dialog --begin 2 2 --backtitle "Raspberry Pi Data - $(date +'%T' )" \
       --yes-label "Refresh" --no-label "Cancel" --timeout 5 \
       --title "DHT11 Sensor Results" --yesno  "$message" 8 30; 
} 

response=0
# Cycle is the response is YES button or the dialog timed out
while [ "$response" == "$YES" ] || [ "$response" == "$TIMEOUT" ]; do
  show_dlg
  response=$? ;# Get the output from the yesno dialog
done
clear

Radio Dialog to Toggle Pi GPIO Pins

A radio dialog allows a user to select one option from a list.

For this example I’m using 3 GPIO pins (0, 2, and 7) and the user can select one of these pins and toggle the output using the Raspberry Pi gpio utility.

The Dialog utility will output the item selected (0, 2 or 7), and the OK/Cancel button code (0=OK,1=Cancel).

#!/usr/bin/bash
#
# dradio1.sh - toggle a GPIO output pin
#
thepin=$(dialog --begin 2 2 --title "Toggle GPIO Pins" \
     --backtitle "Raspberry PI GPIO" --stdout \
     --radiolist "Select Pin:" 10 40 3 \
      0 "GPIO_0 - physical pin 11" off \
      2 "GPIO_2 - physical pin 13" off \
      7 "GPIO_7 - physical pin 7 " off )
clear
# Toggle if OK entered and a pin is selected
if [ "$?" == "0" ] && (( ${#thepin} > 0 )) ; then
  echo "Toggling (wPi) pin: $thepin"
  gpio toggle $thepin
  echo "Pin $thepin is: $(gpio read $thepin)"
fi   

Weather Station Form

A Form dialog can be used text with captions and allow user input to saved. Below is an example of a Weather station with view only and editable data.

#!/usr/bin/bash
#
# dform1.sh - Form to show data, and allow data entry

# Weather Sensor inputs (connect real inputs here)
wspeed="5 kph"
wdir="NW (350)"
wtemp="11 C"

# Show a dialog with viewonly and data entry values, save to a file
dialog --begin 2 2 --ok-label "Save" --backtitle "Pi Weather Station" \
     --title "North Beach" --stdout  \
     --form  "Data at : $(date +'%T' )"  12 65 0 \
	"Wind Speed :"      1 1 "$wspeed"  1 30 0 0 \
	"Wind Direction:"   2 1 "$wdir"    2 30 0 0 \
	"Water Temp:"       3 1 "$wtemp"   3 30 0 0 \
	"Beach Conditions:" 4 1 ""  	   4 30 30 30 \
	"Wildlife:"         5 1 ""  	   5 30 30 30 > beach.txt     
clear

The syntax for the form is:

--form text height width formheight [ label y x item y x flen ilen ]

where: y = line position
       x = position in line
    item = view only or editable data
    flen = field length , 0 = view only
    ilen = input length , 0 = view only

For this example the last 2 items (Beach Conditions and Wildlife) have a field and input length defined to 30 characters so data can be entered into these fields. If the OK button is selected the user entered data is saved to the file beach.txt.

Menu Example

Menuing applications are probably the most useful feature in the Whiptail and Dialog utilities.

Below is an System Information application, that has 3 options: node, disk space, and memory stats. Each of the menu items call a display_dialog function that presents the results of a Bash statement in a message box.

#!/bin/bash
#
# dmenu.sh - Bash Dialog Menu showing some system stats
#

# Display menu results in a msgbox
display_result() {
  dialog --title "$1" \
    --backtitle "System Information" \
    --no-collapse \
    --msgbox "$result" 0 0
}

while true; do
  selection=$(dialog --stdout \
    --backtitle "System Information" \
    --title "Key Features" \
    --clear \
    --cancel-label "Exit" \
    --menu "Please select:" 0 0 4 \
    "1" "Display Node Information" \
    "2" "Display Disk Space" \
    "3" "Display Memory Stats" \
     )
  exit_status=$?
  if [ $exit_status == 1 ] ; then
      clear
      exit
  fi
  case $selection in
    1 )
      result=$(echo "Hostname: $HOSTNAME"; uptime)
      display_result "System Information"
      ;;
    2 )
      result=$(df -h)
      display_result "Disk Space"
      ;;
    3 )
      result=$(vmstat --stats)
      display_result "Memory Stats"
      ;;
  esac
done

Changing Whiptail Default Colors

Whiptail uses the newt graphic library. A NEWT_COLORS variable can be created with custom colors. An example would be:

export NEWT_COLORS='
  window=,red
  border=white,red
  textbox=white,red
  button=black,white'
# to reset the color back to default use:
# unset NEWT_COLORS

A full definition of all the options and colors:

root                  root fg, bg
border                border fg, bg
window                window fg, bg
shadow                shadow fg, bg
title                 title fg, bg
button                button fg, bg
actbutton             active button fg, bg
checkbox              checkbox fg, bg
actcheckbox           active checkbox fg, bg
entry                 entry box fg, bg
label                 label fg, bg
listbox               listbox fg, bg
actlistbox            active listbox fg, bg
textbox               textbox fg, bg
acttextbox            active textbox fg, bg
helpline              help line
roottext              root text
emptyscale            scale full
fullscale             scale empty
disentry              disabled entry fg, bg
compactbutton         compact button fg, bg
actsellistbox         active & sel listbox
sellistbox            selected listbox

bg and fg can be:

color0  or black
color1  or red
color2  or green
color3  or brown
color4  or blue
color5  or magenta
color6  or cyan
color7  or lightgray
color8  or gray
color9  or brightred
color10 or brightgreen
color11 or yellow
color12 or brightblue
color13 or brightmagenta
color14 or brightcyan
color15 or white

Changing Default Dialog Colors

The custom Dialog colors are defined in the file: ~/.dialogrc

To create and edit this file:

dialog --create-rc  ~/.dialogrc
nano $HOME/.dialogrc

Within the ~/.dialogrc file, an important option is:

# Shadow dialog boxes? This also turns on color.
use_shadow = OFF

Dialog supports inline color (this isn’t supported in Whiptail) with the –colors option. Inline colors are defined with by “/Zx”:

ANSI colors:

/Z0 = black
/Z1 = red
/Z2 = green
/Z3 = yellow
/Z4 = blue
/Z5 = magenta
/Z6 = cyan
/Z7 = white

Other options:

/Zb = bold, /ZB = reset bold
/Zr = reverse, /ZR = reset reverse
/Zu = underline, /ZU = reset underline
/Zn = restore settings to normal

An example:

$ msg="Temperature: \Zb\Z3 28 \Zn Deg C"
$ dialog --title "Outside" --colors --msgbox  "$msg" 0 0 ; clear

Final Comments

I’m a big fan of Zenity and YAD X-Window dialog tools, and I found that it wasn’t a big transition to use Whiptail and Dialog.

It’s important to note that the Dialog option –stdout is needed if you want to pass the Dialog output to a variable. Passing the output from Whiptail is a little trickier, use: 3>&2 2>&1 1>&3

Xonsh: a Python and Bash Shell

Xonsh is a Python-powered, cross-platform shell language and command prompt, that works on all major systems including Linux, OSX, and Windows.

Bash scripts are fast and effective for small or batch applications. One of the limitations in Bash is it’s handling of math functions and floating point numbers.

Python is super popular for new programmers and it has a huge library of available functions.

For Raspberry Pi users Xonsh can offer a lot of opportunities to write some extremely lean scripts. Python can be used to connect to 3rd party devices and sensors and 1 line of Bash can be used for dialogs.

In this blog I will introduce Xonsh with some Raspberry Pi examples.

Getting Started

See the Xonsh documentation for installation directions on your specific system. For installation on Raspberry Pi/Ubuntu/Debian enter:

sudo apt install xonsh

To run the Xonsh, simply enter: xonsh

Out of the box Xonsh offers a configuration wizard and a tutorial.

Using Python

Python code can entered directly at the command line. The version of Python will depend on what is loaded on the base system. To check your version:

$ import sys
$ sys.version
'3.9.2 (default, Mar 12 2021, 04:06:34) \n[GCC 10.2.1 20210110]'

Like the interactive Python interface, print statements are not required to see the output:

$ 5 + 6
11
$ a=4;b=7
$ a+b
11
$ msg1="hi"
$ msg1 + " world"
'hi world'

Using Bash

Xonsh uses Python first, so an example with ls (the Bash list command):

$ # This first ls is used as a Bash list command
$ ls
Adafruit_DHT           LICENSE      README.md  dist      setup.py
Adafruit_DHT.egg-info  MANIFEST.in  build      examples  source
$ ls="this is a variable"
$ # Xonsh show the variable ls
$ ls
'this is a variable'

In the above example ls is first used as the Bash list command, but if a variable is defined with the same name, the variable is referenced.

Xonsh processes a Bash statement as a single line. This means that:

  • Bash for/while/if statements need to be all on one line
    • Remember to use white space between characters
  • Line extensions (with a “\”) are not supported
  • Bash functions are not supported.
    • You can write the function in Python instead of Bash

Using Python in Bash

Python statements are using within Bash by: @(Python statements). Below are two examples of using Python in Bash:

$ import sys
$ echo @(sys.version)
3.9.2 (default, Mar 12 2021, 04:06:34)
[GCC 10.2.1 20210110]
$ echo @("Answer=" +str(5+6))
Answer=11

Using Bash in Python

Bash variables can be used directly in Python, for example:

$ #Use Bash date and pass it to Python
$ now=$(date)
$ print("Now is: " + now)
Now is: Thu 24 Mar 2022 03:59:32 PM EDT

Pi Example: Show CPU Temp

Showing the Raspberry Pi’s CPU temperature is a simple example to show how Bash and Python can be used effectively together. Pi’s CPU temperature is stored internally as milli-DegC (1,000 times the degrees in Celsius). Floating point math in Bash is awkward, so this can be done in Python.

Below is a Xonsh script to read the CPU temperature, convert the value to one decimal point, and then present the result in a Bash/Zenity dialog:

#!/bin/xonsh
#
# temp2dlg.sh - use xonsh to get Pi temperature and show in a dialog
#

# Get the temperature with a Bash "cat" and variable
mtemp=$(cat /sys/class/thermal/thermal_zone0/temp)

# Use Python to convert a string of milli-DegC to DegC with 1 decimal
btemp=str(round(int(mtemp)/1000,1)) + " Deg C"
# Add some formatting
btemp="<span font='32' foreground='red'>" + btemp + "</span>"

# Use a Bash Zenity dialog to show the result
zenity --info --text=@(btemp) --title=Raspberry_PI_CPU_Temp  --width=300 &

To make the script executable and then run it:

$ chmod +x temp2dlg.sh
$ ./temp2dlg.sh

Raspberry Pi DHT11 Sensor Value

For the next project I used a DHT11 temperature/humidity sensor.

My goal for this project was to do a snapshot and present the data in a list dialog.

The code to read and present the data was only 7 lines. The Bash variable is passed between Bash statements just like a Python variable would be passed with the @(Python_statement) syntax. Really the only ugly part of the code was the long Zenity statement.

#!/bin/xonsh
#
# dht11_dlg.sh - using xonsh show DHT11 sensor data on a dialog
#
import time
import Adafruit_DHT
sensor=Adafruit_DHT.DHT11
gpio=17

humidity, temperature = Adafruit_DHT.read_retry(sensor, gpio)

thetime=$(date)

zenity --list --title=DHT11_Sensor_Data  --text=@(thetime) --column=Sensor --column=Value --column=Units Humidity @(humidity) "%" Temperature  @(temperature)  "Deg C"

Issues with Shells

Working between different shells and sub-shells can be a little confusing. I found that I occasionally got confused which shell I was working in. The ps command would tell me if xonsh is running:

I was able to pass an Xonsh script to a Bash script without any issues, but I found that for certain operations I needed to manually kill a Xonsh shell.

Below is an example using the DHT11 sensors, and the YAD command line dialog tool (install by: sudo apt install yad). This example had a Python function (show_data) that cycled every 2 seconds and piped the new sensor data to the dialog. Unfortunately the Xonsh shell was still running even after the dialog closed so I needed to use the pkill utility to terminate xonsh .

#!/bin/xonsh
#
# dht11_data.sh - using xonsh show DHT11 sensor data on a dialog
#
import time
import Adafruit_DHT
sensor=Adafruit_DHT.DHT11

gpio=17
# show_data - format sensor data for YAD multi-process bar format
def show_data():
	while True:
		time.sleep(2)
		humidity, temperature = Adafruit_DHT.read_retry(sensor, gpio)
		echo @("1#" + str(temperature) + " Deg C")
		echo @("1:" + str(temperature))
		echo @("2:#" + str(humidity) + " %")
		echo @("2:" + str(humidity))

echo "Showing realtime data..."
@(show_data)  | yad  --multi-progress --bar=Temperature --bar=Humidity --title=DHT11_Sensor_Data 
# exit and ensure that the subprocess is killed
echo "Exit and kill subprocess if running..."
pkill -f xonsh &

Summary

Xonsh has a lot of potential for users looking for simple scripting solutions.

For myself I’ll probably stick to either Bash or Python solutions, but I like that I have other options.

TinyGo: A Go Compiler for Micro-controllers

For micro-controllers like the Arduino or the BBC Micro:bit there are a choice of different programming languages, and now Go (or Golang) can be used for embedded solutions.

Go is an open source language that was developed and supported by Google. Go is a compiled language that runs on MS-Windows, MacOS and Linux.

The TinyGo project implements the exact same Go programming language, however TinyGo uses a different compiler that creates applications on a variety of different micro-controllers.

In this blog I wanted to document some of my notes on building an apps that ran on Arduino and a BBC Micro:bit hardware.

Some key take-aways are:

  • TinyGo is easy to install and flash packages to a micro-controller.
  • The same Go program can potentially run on totally different controllers (some comments on this below).
  • The TinyGo project is still in development so it’s missing some common libraries, full Wifi support being the most important one.

Getting Started

For your specific installation requirements see: https://tinygo.org/getting-started/install/

The typical TinyGo program will have three sections: 1) a definition of libraries used, 2) a setup of hardware and variables, and 3) an infinite loop to run some actions:

Arduino Uno 7 Segment Display Project

For a first project I wanted to show some numbers on a 7 segment display with an Arduino Uno.

The TinyGo site has a list of supported devices: https://tinygo.org/docs/reference/devices/. It’s important to realize that not all the libraries are support on all micro-controllers. There are examples for these drivers at: https://github.com/tinygo-org/drivers/tree/release/examples.

Unfortunately there aren’t examples for all the drivers, so in some cases I needed to look at the source code and reference examples done in Arduino C.

Below is the Go source for a program that will setup the tm1637 segment display on :

//
// Example program to write to a 7 segment display
//
package main

import (
    "machine"
    "time"
    "tinygo.org/x/drivers/tm1637"
)

func main() {
    // Define hardware, clk on pin D2, di (data) on pin D3
    mydisplay := tm1637.New(machine.D2, machine.D3, 7) 

    println("Starting...")
    val := 1

    for {       
        val = val + 1
        print("val: ")
        println(val)
        mydisplay.ClearDisplay()
        mydisplay.DisplayNumber(int16(val))
        time.Sleep(time.Millisecond * 1000)
    }
}

To run this example I needed to create a new directory with the source, initialize the project, get libraries and then build/flash to the controller:

# make a directory for my project
mkdir tm1637
cd tm1637
# initialize a Go project, this project is: tm1637_uno
go mod init tm1637_uno
# get tinygo.org/x/drivers/tm1637 for this app
go get tinygo.org/x/drivers/tm1637
# tidy up project file with tinygo.org/x/drivers/tm1637
go mod tidy
# build and flash project to Arduino 
tinygo flash  -target=arduino tm1637_uno.go  

There are some options on the tinygo compiler, for this example flash -target=arduino will download the application to any Arduino hardware (Uno, Mega, Nano etc.). Tinygo will find the controller so you don’t need to specific a port, however if you have multiple Arduino modules connected you can use the -port option to specify a port.

USB Serial Output

Typically you include USB Serial print statements for debugging. This output can be viewed using a number of different packages, like the Arduino IDE monitor option.

If you are working in a Linux shell, a couple of lines of bash can be used:

# in Linux the default port is: /dev/ttyACM0
# the Arduino Uno default speed is 9600
stty -F  9600 
while read line < /dev/ttyACM0; do echo "$line"; done

Another option is to write a Go USB Serial program. This option has the advantage in that a terminal based program can be used to create custom views of the micro-controller data (I’ll do this later and add some bar charts).

Below is the Go source code for a USB serial interface:

//
// USB Serial program to output data from a Micro-controller
//
package main

import "log"
import "github.com/tarm/serial"
import	"bufio"
// adjust port and speed for your setup
const SERIAL_PORT_NAME = "/dev/ttyACM0"
const SERIAL_PORT_BAUD = 9600

func main() {
	conf := &serial.Config{Name: SERIAL_PORT_NAME, Baud: SERIAL_PORT_BAUD}
	ser, err := serial.OpenPort(conf)
	if err != nil {
		log.Fatalf("serial.Open: %v", err)
	}
	scanner := bufio.NewScanner(ser)
	for scanner.Scan() {
          println(scanner.Text())
	}
	if scanner.Err() != nil {
	  log.Fatal(err)
	}
}

To build this project (source file is: main.go), the standard Go (not TinyGo) compiler is used :

# create a source directory and put main.go into this directory
mkdir USB_Serial
cd USB_Serial
# initialize the project
go mod init main
# get the serial library
go get github.com/tarm/serial
go mod tidy
# build the file, call the executable: usbserial
go build -o usbserial main.go
# run the file and see some sample output
./usbserial

val: 918
val: 919
val: 920
...

Micro:bit Sensor Project

For my next project I wanted to do something more complex. I connected to a BME280 temperature/pressure/humidity sensor and presented the results on 16×2 LCD display, and on a USB serial connection to a laptop.

Both BME280 sensor and the 16×2 LCD display used the I2C bus, so I used a small breadboard to connect the SDA and SCL pins together. It’s important to note that the Micro:bit only has 3.3V power so 5V LCD display units won’t work. My LCD display uses the hd44780i2c driver.

My project code was as follows:

//
// project1.go - show sensor data on a 2x16 LCD Display
//
package main

import (
	"machine"
	"strconv"
	"time"
	"tinygo.org/x/drivers/bme280"
    "tinygo.org/x/drivers/hd44780i2c"
)

func main() {
      // setup sensors
	  machine.I2C0.Configure(machine.I2CConfig{})
	  sensor := bme280.New(machine.I2C0)
	  sensor.Configure()

	connected := sensor.Connected()
	if !connected {
		println("BME280 not detected")
	}
	println("BME280 detected")
        // setup 2x16 screen
        mydisplay := hd44780i2c.New(machine.I2C0, 0x3F)
        mydisplay.Configure(hd44780i2c.Config{Width:16 , Height:2,})
        mydisplay.CursorOn(false)

	for {
		mtemp, _ := sensor.ReadTemperature()
        temp := int(mtemp/1000)

		mpress, _ := sensor.ReadPressure()
		press :=int(mpress/100000)

		mhum, _ := sensor.ReadHumidity()
        hum := int(mhum/100)

        mydisplay.ClearDisplay()
		msg :=  strconv.Itoa(temp) + " C  " + strconv.Itoa(press) + " hPa\n" + strconv.Itoa(hum) + " %"
        println(temp, press, hum)
        mydisplay.Print( []byte(msg) )
		time.Sleep(2 * time.Second)
	}
}

The serial output was sent as integers with no labels or units, I did this for the next step and that was to create a custom terminal program.

Terminal Output

There are a number of different options on how to present the serial output.

I wanted to keep things basic so I used the earlier USB serial code and I included some ANSI terminal codes.

Below is the Go source. I added some ANSI constants for clearing, resetting and color definitions.

The screen will refresh every time some new data is received from the Microbit.

package main

import "fmt"
import "log"
import "strings"
import "github.com/tarm/serial"
import "bufio"
import "strconv"

const SERIAL_PORT_NAME = "/dev/ttyACM0"
const SERIAL_PORT_BAUD = 115200

const clear = "\033[2J"
const reset = "\033[0m"
const uline = "\033[4m"
const nouline = "\033[24m"

// define some foreground colors
const red = "\033[38;5;1m"
const green = "\033[38;5;10m"
const magenta = "\033[38;5;13m"
const purple = "\033[38;5;5m"
const yellow = "\033[38;5;11m"

// Use ANSI color and fill character to show scaled bars with values 
func show_bar(inval string, label string, units string, toprange int, scale int, color string) { 
    bar1, _ := strconv.Atoi(inval) ; // convert USB serial text to int
    bar1 = bar1 * 50 / toprange    ; // scale string input to bar
    bar2 := scale - bar1
    print(color)
    fmt.Printf("%15s %s%s %s %s\n\n", label, strings.Repeat("█",bar1), strings.Repeat("░",bar2), inval, units)

}

func main() {
    conf := &serial.Config{Name: SERIAL_PORT_NAME, Baud: SERIAL_PORT_BAUD}
    ser, err := serial.OpenPort(conf)
    if err != nil {
        log.Fatalf("serial.Open: %v", err)
    }
    scanner := bufio.NewScanner(ser)
    for scanner.Scan() {
        println(clear, uline)
        println(" Micro:Bit Sensor Values ",nouline,"\n")
        values := strings.Split(scanner.Text(), " ")

	show_bar(values[0],"Temperature:","C", 40, 50, magenta)
        show_bar(values[1],"Pressure:","kPa", 1020, 50, green)
        show_bar(values[2],"Humidity:","%", 100, 50, yellow)
        println(reset) ; // Put terminal back to reset mode
    }
    if scanner.Err() != nil {
        log.Fatal(err)
    }
}

If you are looking at doing some more creative terminal programs there are some great libraries, for example: https://github.com/gizak/termui

Same Code on Different Modules

When I looked at running the same Go code on different modules I noticed:

  • Different hardware had different machine libraries, for example Microbit defines pins as P1, P2… , but Arduino defines pins as D1, D2 …. This will unfortunately means the same code can’t run directly.
  • I2C bus interfaces use SCL and SDA definitions which are uniform for all controllers, so my 2nd project would run on probably all the platforms without any code changes.
  • Libraries are hardware dependent, so a library that works on an Arduino may not work on other devices

Final Comments

Because TinyGo is still in development I found it a little challenging because of the lack of documentation. I especially missed having a Wifi library.

For Arduino I would definitely stick to the default C/C# interface, however for the Microbit C/C# isn’t a good option so I can see using TinyGo.

If you are looking at learning Go and you want to start with some small projects TinyGo might be a good option.

Gemini Protocol for Lightweight Internet Apps

The Gemini protocol is a very simple and light Internet protocol.

Unlike HTML files that contain layers of tags, style sheets and Javascript, a Gemini document is a simple readable document.

In this blog I wanted to document:

  • Gemini servers and clients using only 1 line of Bash
  • How to use large ASCII text in Gemini documents
  • How to create simple bar charts
  • How to create Gemini CGI apps in Python

Getting Started

A Gemini document only supports a few statements and graphic images such as JPEG or PNG are not supported.

An example Gemini document with the common formatting options:

# Heading level 1 (H1) 
## Heading level 2 (H2)
### Heading level 3 (H3)

=> testpage.gmi A link to another page.

> This line will show as a block-quote. 

A list of items
* This is the first list item.
* This is another list item.

```
Code or ASCII Block 
   _   ___  ___ ___ ___   _____       _   
  /_\ / __|/ __|_ _|_ _| |_   _|__ __| |_ 
 / _ \\__ \ (__ | | | |    | |/ -_|_-<  _|
/_/ \_\___/\___|___|___|   |_|\___/__/\__|

```

Within a Gemini browser this file would look like:

Content Type

The content type is used by browsers and applications to determine how to manage the requested file.

Typically the content type is managed by server, for example a web server will send a HTTP/1.0 200 OK prior to sending the HTML file.

For the Gemini protocol the content type is: 20 text/gemini . Depending on the Gemini server the user may need to be add the content type manually. (More about this in Bash and CGI servers).

Simple Bash Gemini Servers and Clients

For basic testing a one line Bash statement can be used for custom Gemini servers and clients.

The Gemini protocol typically uses SSL (Secure Sockets Layer) and TLS (Transport Layer Security) encryption so the Bash ncat utility is needed (Note: the nc command does not support SSL).

Below is an example of single Gemini request:

The Gemini server is defined by using the -l , listen option. When a client requests data, the cat statement is used with a pipe (|) to output the file testpage.gmi.

The Gemini client echo’s a “from PC” message with its request, this helps identify which client is requesting data.

A simple Bash ncat statement doesn’t manage the content type so a “20 text/gemini” line is added to the top of the test page.

Dynamic Bash Data

In the earlier example the Bash server is only supporting 1 request then exiting.

A while loop can be added to pass a Bash script to the ncat statement. Below is an example of Bash script (showdata.sh) that show CPU data using the vmstat utility:

#!/bin/bash
#
# showdata.sh - Output data for Gemini Bash Server 
#
echo "20 text/gemini"
echo  " "
echo "#VMSTAT"
echo  " "
date +"%T"
echo  " "
# set Gemini formating for ASCII
echo  "\`\`\`"
#vmstat
top -n 1
echo  "\`\`\`"

To make the script executable use the command: chmod +x showdata.sh

The Bash command to run this script as a Gemini server is:

while true; do ./showdata.sh | ncat  -l -p 1965 --ssl; done

The earlier Bash Gemini client command can be used, or a Gemini browser/client app can be used. The handling of SSL/TLS encryption will vary with the client app. I used the very basic Zain app (https://gitgud.io/sathariel/zain) :

(Note: for the Zain client I needed to load tcl/tls, sudo apt-get install -y tcl-tls)

Using a 1 line Bash Gemini server is great for basic testing but I wouldn’t recommend if you want to connect to variety of different Gemini client applications.

Large ASCII Text

Gemini documents don’t support different font sizes, a workaround is to use the figlet tool to generate multi-line text strings. Figlet is installed on Ubuntu/Debian/Raspbian by:

sudo apt install figlet

Figlet has a number of font styles that use 2-5 line height characters:

When using ASCII headings the Gemini code formatting option should be used, and this has three backticks (“`) before and after the headings.

The earlier example can be modified to have ASCII headings:

#!/bin/bash
#
# showdata2.sh - Output Large Headings to a Gemini Bash Server 
#
echo "20 text/gemini"
echo  " "
echo  "\`\`\`"
# Generate large text 
figlet -f standard "Raspberry Pi"
figlet -f small -m 2  $(date +"%T")
# show CPU stats
vmstat
echo  "\`\`\`"

Bar Charts

Horizontal ASCII bar charts can be created by using the printf statement with different ASCII fill characters. For example:

# show a label with bar and value text 
#
label="temp"; val=20;
bar="printf '█%.0s' {1..$val}" ; 
printf '\n%-5s ' $label; eval $bar
printf '░%.0s' {1..5} ;
printf ' 50 C\n'

temp  ████████████████████░░░░░ 50 C

This bar logic can be using in a Raspberry Pi Stats page that looks at idle time and free space on the SD card:

#!/bin/bash
#
# pi_stats.sh - Output Pi Stats as a Gemini Page
#
echo "20 text/gemini"
echo  " "
# Put the rest of the Gemini document into code block mode
echo  "\`\`\`"
# Generate large text 
figlet -f standard "Pi Stats"

# Show the time
echo "Time: $(date +'%T')"
echo ""

# Get idle time, scale 0-50
idle=$(top -n 1 | grep id | awk '{print $8}')
barsize=$(echo $idle | awk '{printf "%0.f" , $1/2}')
thebar="printf '█%.0s' {1..$barsize}"
graysize=$(expr 50 - $barsize)
thegray="printf '░%.0s' {1..$graysize}"
printf 'Idle Time  '; eval $thebar; eval $thegray ; echo " $idle %"
echo  ""

# Get free space on SD card, scale 0-50
freesp=$(df | grep root | awk '{printf "%0.f", $5/2}')
barsize=$(echo $freesp | awk '{printf "%0.f" , $1/2}')
thebar="printf '█%.0s' {1..$barsize}"
graysize=$(expr 50 - $barsize)
thegray="printf '░%.0s' {1..$graysize}"
printf 'Free Space '; eval $thebar; eval $thegray ; echo " $freesp %"

echo  "\`\`\`"

To run this page use

while true; do ./pi_stats.sh | ncat  -l -p 1965 --ssl; done

Python CGI Pages

There are a number of good Gemini servers, for my testing I used the Python based Jetforce server, it is installed by:

pip install jetforce

To run the Jetforce server it is important to define a home directory, the allowable hosts that can connect (0.0.0.0 is all IP4 nodes) and the server’s hostname:

# Start the jetforce Gemini server for all IP4 hosts
jetforce --dir /home/pi/temp --host "0.0.0.0" --hostname 192.168.0.105 &
# Start jetforce without hardcoding hostname
# jetforce --dir /home/pi/temp --host "0.0.0.0" --hostname $(hostname -I) &

By default CGI (Common Gateway Interface) files are defined in the directory cgi-bin which is under the home directory.

Unlike a one-line Bash server, Jetforce server will pass environment variables like QUERY_STRING and host and remote connection information.

CGI programs can be written in a variety of programming languages. For this example I wanted to pass Raspberry Pi BME280 temperature, pressure and humidity sensor information to a Gemini CGI page.

The CGI program was written in Python and I installed a bme280 and figlet library:

pip install RPi.bme280
pip install pyfiglet

The Python script (sersor2gmi.py) outputs a Gemini content type, a Figlet title and then the sensor data values:

#!//usr/bin/python3
#
# sersor2gmi.py - send BME280 sensor data to a Gemini page
#
import smbus2
import bme280
import pyfiglet

# find device address via: i2cdetect -y 1
port = 1
address = 0x77
bus = smbus2.SMBus(port)

calibration_params = bme280.load_calibration_params(bus, address)

# the sample method returns a compensated_reading object
data = bme280.sample(bus, address, calibration_params)

# Output a Gemini page
print("20 text/gemini") #Note: Some Gemini CGI servers may do this
print("```") # use code mode
#print(pyfiglet.figlet_format("Pi BME280 Data", font = "small"))
print(pyfiglet.figlet_format("Pi BME 280 Data", font = "small"))

print("Temperature:" + "{:5.1f}".format(data.temperature) + " C" )
print("Pressure:   " + "{:5.1f}".format(data.pressure) + " kPa" )
print("Humidity:   " + "{:5.1f}".format(data.humidity) + " %" )
print("```") 

This file was added to the cgi-bin directory and it is made executeable (chmod +x sersor2gmi.py).

Below is the output seen in the Lagrange Gemini browser:

Final Comments

There are a variety of Gemini browsers for Windows, Android, Mac and Linux so if you’re looking for a quick and dirty internet solution Gemini might be a good solution.

I like how Gemini documents are totally readable, I can’t say the same for most web pages.

The one thing that I missed with Gemini pages is the ability of show nice charts, text based bars work for simple stuff but doing text based line charts is a little too ugly for me.

SMath – A Free MathCAD Alternative

For engineering, physics and math user MathCAD is a great package for theoretical calculations and reports. Unfortunately MathCAD is rather expensive for students and causal users, and it also isn’t support in Linux.

SMath Studio is a free alternative that supports MathCAD files and it works in Linux.

In this blog I wanted to highlight some of my notes in using SMath Studio. Some of the key take away points are:

  • SMath offers a Worksheet approach that is far superior to Excel/LibreOffice for math problems.
    • Equations and matrices are shown like they were hand drawn
    • Units are included in the equations, and unit conversion is automatic
    • easy to read IF/THEN/ELSE, WHILE and FOR LOOP can be using within a Worksheet
  • SMath programming is done visually, (as opposed to writing code like Python or Matlab/Octave).
  • SMath support basic plotting but it is weak compared to other packages like Matplotlib.
  • SMath is good for visual reports, but Python would be a better solution for managing large amounts of data or when statistical calculations are needed.

Getting Started

SMath Studio is supported on Window, MacOS and Linux, see https://en.smath.com/ for installation files.

For Linux user the Mono interface will need to be loaded:

sudo apt install mono-devel
# then download : SMath Studio Desktop for Mono  
# extract to a folder
# to run SMath Studio in Linux:
./smathstudio_desktop_mono

The SMath Studio has a side panel of common functions and symbols. All the functions are listed in the fx dialog.

Freehand Calculations

Smath allow users to create complex freehand calculations using the side panel and some keyboard shortcuts. The arrow key moves the cursor to different sections of a large expression that need to be modified.

Variables and Units

Sheets are calculated from top to bottom, so it is important to define a variable before it is used. A variable is defined with a “:=“, the equal sign (“=“) is used to show the value of a variable.

For equations where units are used a dropdown of available units is shown. The default system is in metric so the conversion is also shown. Units will appear in blue.

The system will automatically manage the conversion between units, and there are some different presentations to show the steps in the conversion, fractions, or just the numeric value.

Show Solutions

Pages can be laid out to create clear presentations with variables that are passed to equations. Like with the units, equations can be shown with their intermediate values. Below is an example that shows a venturi flow rate calculation with the intermediate values and the conversion between Newtons (N), kilograms (kg) and meters (m) to a flow rate of cubic meters/sec.

Plotting and Advanced Functions

Below is a worksheet that allows a user to play with a 2nd order equation. SMath offers some graphical widgets such as: analog sliders, dropdown lists, up/down selector and toggle buttons. For this example I used three slider widgets allow users to change the constants (a,b,c). A plot is used to visually check if the equation has any X-intercepts. The SMath solve function will find X-intercepts. A small if statement is used to check if there are any intercepts. Finally an integration function calculates the area between the limits.

File I/O

SMath supports import/export data functions for: CSV, PDF, XLS, ODF, ODS and a few other. SQLite3 databases are also supported

Below is an example worksheet where a CSV file is imported, cleaned and then exported out to a new CSV file. The data is read in with the importData function to a matrix variable (rawdata).

A for loop iterates through each row data, and an if statement checks a row for positive values. The stack function appends valid rows of data to the cleandata matrix. The last step is to use the exportData_CSV function to write out the cleandata matrix.

Automating SMath

The Windows version of SMath Studio offers a number of command line options that can be used to save and print worksheets. Unfortunately these options don’t exist in the Linux (Mono) version.

As a work around it’s possible to use some keyboard automation tool in Linux to offer some interesting solutions. Below is a script to:

  • open SMath with a user defined worksheet
  • grab the SMath window focus
  • Save as PDF
  • Quit SMath
#!/usr/bin/bas
#
# auto_smath.sh - open
#
myfile="quad" ; # prefix of smath file
rm $myfile.pdf ; # remove existing PDF

# Open smath with a set file, quiet output 
./smathstudio_desktop_mono $myfile.sm  &
sleep 3

# Set focus to SMath
wmctrl -a "SMath Studio"

# Send keystroke to save as PDF
xdotool key alt+F 
sleep 1
for i in {1..4}
do
  xdotool key Down 
  sleep 1
done
xdotool key Return
sleep 1
xdotool key Tab 
sleep 1
xdotool key p
sleep 1
xdotool key Return
sleep 3 ; # Give time to save PDF before exiting

# Exit SMath
xdotool key alt+F4
echo "Done Running $myfile.sm ... saved to $myfile.pdf"

This script uses the wmctrl is get and manage the window focus, and xdotools to send keystrokes. They can be installed by:

sudo apt install wmctrl
sudo apt install xdotool

Some Final Thoughts

I wish I had SMath when I was in university it would have made things so much easier and reduced a lot of manual calculation errors.

It takes a few hours to get used to the keyboard short cuts and formatting.

A few times I forgot about the fact that sheets are calculated from top to bottom and this caused me to get some strange errors. If the software locks up use the following line to kill things:

ps -e | grep mono | awk '{system("sudo kill " $1 "  1>&-")'}

Memcached: fast lightweight network cache

For many projects an SQL database is overkill for simple storage of values. As an alternate solution there are a number of excellent distributed in-memory caching systems that can be used.

Two of the most popular in-memory caching systems are Redis and Memcached.

I’m big fan of Redis, and I’ve enjoyed doing projects with it. Redis offer some awesome speed with a small footprint, and it has many features that make it even superior to a messaging system like MQTT.

However if you’re looking for something 100% dead simple you should take a look a memcached, it has a super simple setup and with only a dozen commands so you’ll get up and running in no time.

There are API’s in all the common programming languages. It only takes 1 line of Bash to read or write to memchached.

Getting Started

Memached can be installed on all major OS’s. To install it on Ubuntu/Raspberry Pi:

sudo apt-get install memcached

If you’re using Docker there are some lightweight memcached images (89MB) that can be used. To run the memcached docker image the –net host option should be used:

$ sudo docker run -it --net host memcached

The -it (interactive) option could be useful if there are any errors kicked out.

See the man pages for a full description of memcached options.

Telnet can be used to enter manual enter commands (port 11211 is the default):

$ telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
version
VERSION 1.6.12

lru_crawler metadump all
key=mynum exp=-1 la=1642363105 cas=2 fetch=no cls=1 size=66
END
quit
Connection closed by foreign host.

For more information on the commands and Telnet.

Bash Interface to Memcached

There are lot of help guides for other Python, PHP, GO etc., but not a lot on how to use Bash.

The Bash nc command can be used to read and write to socket. The -q option will close the socket after 0 seconds (after the echo command is sent). The default port is 11211 but this can be changed along with adding some security options. The first example is to use the stats command:

$ echo "stats" | nc -q 0  192.168.0.120 11211 
STAT pid 1
STAT uptime 722
STAT time 1642359428
STAT version 1.6.12
...
END

$ # Find the number of current key items
$ echo "stats" | nc -q 0  192.168.0.120 11211 | grep curr_items
STAT curr_items 1

Set/Get a Key and Value

To set an in-memory key-value store, the syntax is:

set mykey <flags> <ttl> <size>
value

The flags option is typical set to 0. The ttl “time to live” is in seconds, a ttl of 0 is indefinite.The size of the value also needs to be define. (This is taken care of in the Python, PHP… libraries). It’s important to note that a newline (\r\n) is required before the value.

To set a key to a variable mynum with a value of 55 and a indefinite time to live:

$ # hard code the arguments in a set key command
$ # Note: echo -e is used to pass the \r\n for new lines
$ echo -e "set mynum 0 0 2 \r\n55\r" | nc -q 0  127.0.0.1 11211
STORED

A more flexible approach would be to pass variables:

$ # Set a new key/value pair, with no flags and indefinite timeout
$ ipp="192.168.0.120 11211"
$ thekey="mykey1"
$ thenum=55
$ numsize=${#thenum}
$ # send command 
$ echo -e "set $thekey 0 0 $numsize\r\n$thenum\r" | nc -q 0  $ipp
STORED

Get a key/value

The get key command returns 3 lines with the value being the 2nd line. Some awk code can be used to parse out just the value.

$ # Get a new key/value pair
$ ipp="192.168.0.120 11211"
$ thekey="mykey1"
$ 
$ echo "get $thekey" | nc -q 0  $ipp
VALUE mykey1 0 2
60
END
$ # Get just the 2nd line with the value
$ echo "get $thekey" | nc -q 0  $ipp | awk '{if (NR == 2) print $0}'
55
$ 
$ # Store the result in a variable
$ mykey1=$(echo "get $thekey" | nc -q 0  $ipp | awk '{if (NR == 2) print $0}')
$ echo "mykey1 = $mykey1"
mykey1 = 55

Increment/Decrement a Value

The inc / decr commands will increase or decrease a numeric stored key value by a defined amount:

$ #incr/decr a value's number
#
ipp="192.168.0.120 11211"
thekey="mykey1"

# Get starting value
echo "get mynum" | nc -q 0 $ipp | awk '{if (NR == 2) print $0}'
55

thediff=10; # increase the value (only positive values)
# incr and show the value
echo "incr $thekey $thediff" | nc -q 0  $ipp
65

thediff=5; # decrease the value (only positive values)
# incr and show the value
echo "decr $thekey $thediff" | nc -q 0 $ipp
60

Prepending and Appending

Memcached does not have queue or list functionality, if you need this take a look at Redis.

Below is an example script that creates a diagnostic log in a key/value. The append command adds msg text to the end of the overall string.

#!/usr/bin/bash
#
# diagmsg.sh - append msgs to a memcached variable string
#
ipp="192.168.0.120 11211"
msg="1:00 - Base Software Loaded\r"
size=${#msg}

echo -e "set mymsg 0 0 $size\r\n$msg\n\r" | nc -q 0  $ipp

# Create an array of diagnostics
diagmsgs=("2:00 - System Started\r" "2:15 - Getting Data\r" "2:30 - Backing up\r") 
# Append array to value text
for msg in "${diagmsgs[@]}"
do
  size=${#msg}
  #echo "Size: $size $msg";
  echo -e "append mymsg 0 0 $size\r\n$msg\n\r" | nc -q 0  $ipp;
done

# Show the result
echo -e "\nDiagnostic Results\n" 
echo "get mymsg" | nc -q 0 $ipp 

The results of this script would be:

$ bash diagmsg.sh
STORED
STORED
STORED
STORED

Diagnostic Results

VALUE mymsg 0 92
1:00 - Base Software Loaded
2:00 - System Started
2:15 - Getting Data
2:30 - Backing up

END

Final Thought

I played with the Python and PHP library and they were quite easy to use.

By adding a Bash interface to memcache it allows me to use programs like Octave/Matlib where a native interface isn’t available (just use the System call and pass the Bash code).

Sample Code: PHP

PHP example:

<?php
// memc.php - test memcached
//
// Load the PHP memcached component:
//
//    sudo apt-get install php-memcached

$memcache = new Memcached();
$memcache->addServer('localhost', 11211) or die ("Could not connect");

$version = $memcache->getVersion();
var_dump( $version);

$tmp_object = new stdClass;
$tmp_object->str_attr = 'test';
$tmp_object->int_attr = 123;

$memcache->set('key', $tmp_object);
echo "Store data in the cache (data will expire in 10 seconds)<br/>\n";

$get_result = $memcache->get('key');
echo "Data from the cache:<br/>\n";

var_dump($get_result);

?>

Sample Code: Python

Python example:

$ # Install Python memcache library
$ python3 -m pip install pymemcache

$ # Example set/get
$ python3
Python 3.8.10 (default, Nov 26 2021, 20:14:08) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pymemcache.client import base
>>> client = base.Client(('localhost', 11211))
>>> client.set('mykey','teststring')
True
>>> client.get('mykey')
b'teststring'
>>> client.set('mynum',44)
True
>>> client.get('mynum')
b'44'