Bash with MQTT

I’m working on re-purposing an old router. One of my goals is to bring sensor and performance data from the router to a Home Assistant node. The router doesn’t not have a lot of space so I’d prefer to use Bash rather than Python for MQTT communications.

While I was working on the project I wanted to use some simple tools to view the data, unfortunately I wasn’t able to find any information on how to make a good MQTT client in Bash.

This blog documents how I used Bash to show bar charts of MQTT data.

Setup

The first step is to install the Mosquitto command line tools on the OpenWrt router:

# Update the package manager list
opkg update
# Install the MQTT client utiliy
opkg install mosquitto-client-nossl

When I have some time I’d like to look at making some more MQTT Bash scripts that could be used with file input.

The next step is to install the Mosquitto client on my Linux PC and Raspberry Pi:

sudo apt-get install mosquitto-clients

There are a number of MQTT brokers that can be used, both Home Assistant and Node Red have reliable brokers. The Mosquitto broker can also be loaded on a Linux, MacOS and Windows node.

Publish MQTT Data in Bash

The Mosquitto client can be used to both publish and subscribe to MQTT data.

For my router project, I wanted the: temperature (from a USB thermometer), idle time, % used space and available space. Below is the script that passed the data to the mosquitto_pub (publishing) tool:

#!/usr/ash
#
# mqtt_data.sh - send data to MQTT broker
#
# Get Data values
temp=$(/usbstick/temper.sh)
idle=$(vmstat | awk '{ if (NR==3) print $15}')
used=$(df | awk '{if (NR==4) printf "%.f\n", $5 }')
space=$(df | awk '{if (NR==4) printf "%.1f\n", $4/1000 }')

echo "$temp $idle $used $space"

# Publish Data
server="192.168.0.111"
pause=2

mosquitto_pub -h $server -t rtr_temp -m $temp
sleep $pause
mosquitto_pub -h $server -t rtr_idle -m $idle
sleep $pause
mosquitto_pub -h $server -t rtr_used -m $used
sleep $pause
mosquitto_pub -h $server -t rtr_space -m $space

I added a pause between each publish to let me watch the actions.

The cron utility can be used to schedule the running of the script. Once a minute is the fastest time available with cron, so if faster times are needed the script could cycle with a while loop.

Read/Subscribe to MQTT Data

The mosquitto_sub client tool can be used to read or subscribe to the data.

To look at the router points:

$ service="192.168.0.111"
$ mosquitto_sub -h $server -v -t rtr_idle -t rtr_temp -t rtr_used -t rtr_space

rtr_temp 26.25
rtr_idle 100
rtr_used 66
rtr_space 1.1

The -v (–verbose) option show output with the topic names and the values. Topics, -t option, can be put on the command line or passing in as a file.

For single point monitoring the Zenity utility can be used. This utility is typically preloaded on most Linux and Rasp Pi systems. A script to create a progress bar dialog with MQTT data is:

# Send MQTT values to a progress bar
( 
  while :; do
  msg=$(mosquitto_sub -h 192.168.0.111 -C 1 -t rtr_temp) 
  echo $msg
  echo "#$msg  Deg C"
  done
  ) | zenity --progress  --title="Router External Temperature"

Multipoint Progress Dialogs

The Zenity utility can only manage single point progress bars. For multiple point progress bars the YAD (Yet Another Dialog) package can be used.

To install YAD on Raspberry Pi’s and Ubuntu: sudo apt-get install yad

Below is some script that will show multiple points in a YAD dialog:

#!/usr/bash
#
# mqtt_bars.sh - Show multiple MQTT Topics on a Dialog
#
server="192.168.0.111"
topics=("rtr_idle" "rtr_temp" "rtr_used" )
scale=(100 40 100 )
units=( '%' 'degC' '%'  )
title="Router MQTT Points"

#Build topic and yad strings
yadstr=" "
for i in ${topics[@]}; do
  topstr="$topstr -t $i"
  yadstr="$yadstr --bar=$i"
done

echo "Press [CTRL+C] to stop..."
# Cycle thru 1 message at a time to YAD 
(
while : 
do 
  msg=$(mosquitto_sub -h $server -v -C 1 $topstr) 
  IFS=' ' read -a data <<< "$msg" 
  # match returned msg to order of bars, write value/label
  for i in "${!topics[@]} "
  do 
    if [ "${data[0]}" = "${topics[i]}" ]
    then 
      let j=i+1 ; # YAD indices start at 1
      # Rescale bar to defined scale
      barsize=$(bc <<< "${data[1]}*100/${scale[i]}")
      #barsize=100
      echo "$j:$barsize"
      echo  "$j:#${data[1]} ${units[i]}"
  fi
  done
done 
)  | yad --multi-progress $yadstr --title $title

Final Comments

There are probably lots of good 3rd party tools like Gnuplot that could be connected to mosquitto_sub to show real charts.

Bash Bar Charts for Text and Web Pages

This blog documents my notes on creating Bash text bar charts. The key topics are:

  • Create a simple horizontal bars (3 lines)
  • Create dynamic bars (~25 lines for an array of data values)
  • Add support for piped and command line data
  • Include options for: help, resizing, titles
  • Add colour and HTML support

A Horizontal Bar

To create a static bar an printf statements can be used. The seq {0..10} can be used to repeat an ASCII █ fill character 10 times.

$ printf 'LABEL: ' ; \
  printf '█%.0s' {1..10} ; \
  printf ' 10\n'

LABEL: ████████████████████ 10

Unfortunately the printf statement has some limitations on variable substitution. A simple workaround is to create a string and then eval it:

$ label="temp"; val=20;
$ bar="printf '█%.0s' {1..$val}" ; 
$ printf '\n%-5s ' $label; eval $bar ; printf ' %d\n' $val

temp  ████████████████████ 20

Coloured Bars

The tput setaf command can change the foreground, and tput setab is used for background colours. Colour codes are:

tput setab [1-7] # Set the background colour using ANSI escape
tput setaf [1-7] # Set the foreground colour using ANSI escape

Num  Colour    #define         R G B

0    black     COLOR_BLACK     0,0,0
1    red       COLOR_RED       1,0,0
2    green     COLOR_GREEN     0,1,0
3    yellow    COLOR_YELLOW    1,1,0
4    blue      COLOR_BLUE      0,0,1
5    magenta   COLOR_MAGENTA   1,0,1
6    cyan      COLOR_CYAN      0,1,1
7    white     COLOR_WHITE     1,1,1

To reset colours back to the defaults use: tput sgr0

An example to print a red bar and a stack of bars:

$ printf '\nLABEL: ' ; \
   tput setaf 1 ;\
   printf '█%.0s' {1..10} ; \
   printf ' 10\n'

LABEL: ██████████ 10
$ printf '\n 3 Stacked Bars: ' ; \
   tput setaf 1 ;\
   printf '█%.0s' {1..10} ; \
   tput setaf 2 ;\
   printf '█%.0s' {1..8} ; \
   tput setaf 4 ;\
   printf '█%.0s' {1..3} ; \
   printf ' 10+8+3=21\n'

 3 Stacked Bars: █████████████████████ 10+8+3=21

Dynamic Bars

The next step is to create a script that dynamically updates the bars. The tput clear command will clear the terminal screen keep the data and bars in the same location. The script below will dynamically show the CPU temperature, idle time and 2 random values with a 10 second update time.

#!/bin/bash
# 
# cpu_bars.sh - Show new data every 10 seconds
#
while :; do
    # Get data values
    CPUtemp=$(sensors | grep CPU | awk '{print substr($2,2,4)}')
    CPUidle=$(iostat | awk '{if (NR==4) print $6}')
    Random1=$((1+ $RANDOM % 100))
    Random2=$((1+ $RANDOM % 100))

    labels=( CPUtemp CPUidle Random1 Random2)
    values=( $CPUtemp $CPUidle $Random1 $Random2)
    units=( degC % psi mm)

    # Show a title
    tput clear
    printf " %10s " "" 
    tput setaf 7; tput smul;
    printf "%s\n\n" "Show CPU Data ($(date +%T'))"
    tput rmul;

    # cycle thru data and show a label, 
    for index in "${!labels[@]}"
    do
          tput setaf $(expr $index + 1); # don't use 0 (black) 
          printf " %10s " "${labels[index]}"
          eval "printf '█%.0s' {1..${values[index]%.*}}"
          printf " %s %s\n\n" ${values[index]} ${units[index]}
    done
    sleep 10
done

This script is run by: bash cpu_bars.sh .Typical output is below.

Bars in HTML

The ANSI colours are not supported in HTML, so instead HTML tags and style properties can be used.

It is important to use <pre> tags for Bash text output. Code to create two static bars in HTML would be:

$ (printf "<h1>A Bar from Bash</h1>\n" 
 printf "<pre><span style='font-size:24px;color:red'}>\n"
 printf 'LABEL1: ' ; printf '█%.0s' {1..10} ; printf ' 10\n'
 printf "</pre></span>\n") > bar1.htm

$ cat bar1.htm
<h1>A Bar from Bash</h1>
<pre><span style='font-size:24px;color:red'}>
LABEL1: ██████████ 10
</pre></span>

The script cpu_webbars.sh creates HTML output for an array of calculated values:

#!/bin/bash
# 
# cpu_webbars.sh - Show bars in HTML
#

# Get data values
CPUtemp=$(sensors | grep CPU | awk '{print substr($2,2,4)}')
CPUidle=$(iostat | awk '{if (NR==4) print $6}')
Random1=$((1+ $RANDOM % 100))
Random2=$((1+ $RANDOM % 100))

labels=( CPUtemp CPUidle Random1 Random2)
values=( $CPUtemp $CPUidle $Random1 $Random2)
units=( degC % psi mm)
colors=(red blue green magenta)

# Show a title
printf "<h1><center>Show CPU Data ($(date '+%T'))</center></h1>\n"
# cycle thru data and show a label, 
for index in "${!labels[@]}"
  do
  printf "<pre><span style='font-size:18px;color: ${colors[index]} '}>\n"
  printf " %10s " "${labels[index]}"
  eval "printf '█%.0s' {1..${values[index]%.*}}"
  printf " %s %s\n\n" ${values[index]} ${units[index]}
  printf "</pre></span>\n"
done

This script can be run and outputted to an file: bash cpu_webbars.sh > test.htm

Once the basic HTML output is working its possible to add headers and footers to make a more complete page:

header.htm > test.htm ; \
cat cpu_webbars.sh >> test.htm ; \
cat footer >> test.htm 

Piping Data to Bars

A script (hbar0.sh) will read the piped data and then create an array (data) of labels and values. The data array is cycled through and labels and bars are shown with a randomized colour:

#!/bin/bash
# hbar0.sh - Read in piped data  and plot bars
#     format: label,value;label2,value2;  and plot bars
#
input=$(< /dev/stdin) ; # read piped data
# remove new lines in files, and check for ";" after data pair 
input=$(echo $input | tr -d '\n')
IFS=';' read -r -a data <<< $input
printf "\n" 
for element in "${data[@]}"pete@lubuntu:~/Writing/Blog/text_bars
do 
  # make at array of each data element
  IFS=',' read -r -a datapt <<< $element
  # add a random color
  tput setaf $((1+ $RANDOM % 7))
  # print the label, bar and value
  printf " %10s " "${datapt##*[0]}"
  bar="printf '█%.0s' {1..${datapt[1]}}"
  eval $bar
  printf " %s\n\n" ${datapt[1]} 
  tput rmso ; # exit color mode   
done

The script can be tested with piped data:

$ echo "temp,33;pressure,44" | bash hbar0.sh

 temp       █████████████████████████████████ 33 

 pressure   ████████████████████████████████████████████ 44 

A data file can also be passed in using the cat command:

$ cat data0.txt 
temp,44;
humidity,33;
pressure,15;
wind spd,33;
wave ht,3;
$ cat data0.txt | bash hbar0.sh

       temp ████████████████████████████████████████████ 44

   humidity █████████████████████████████████ 33

   pressure ███████████████ 15

   wind spd █████████████████████████████████ 33

    wave ht ███ 3

Removing ANSI Colors

Terminal applications use ANSI color codes which unfortunately is not support on Web pages.

ANSI color codes can be removed from files and strings by:

# Strip out ANSI color codes:
cat infile | sed 's/\x1b\[[0-9;]*m//g' > outfile
$ echo "temp,33;pressure,44" | bash hbar0.sh > hbar0.txt
$ cat hbar0.txt

       temp █████████████████████████████████ 33

   pressure ████████████████████████████████████████████ 44

$ cat hbar0.txt | sed 's/\x1b\[[0-9;]*m//g'

       temp █████████████████████████████████ 33

   pressure ████████████████████████████████████████████ 44

A Final App

With the basics in place I was able to create an app that would support scaling, titles, units, custom colour and web output:

$ ./hbars
usage: hbars [data] [option] 
  -h --help     print this usage and exit
  -c --color    set color to all bars (default 7=white)
                 (0-grey,1-red,2=green,3=yellow,4=blue,5=magenta,6=cyan,7=white)
  -p --pretty   add different colors to bars (-c overrides)
  -t --title    top title
  -w --width    max width of bar (default 50)
  -W --Web      make output HTML formatted
  -f --fontsize fontsize for web output (default 24)

 examples:
   echo 'temp,33,C;pressure,14,psi' | ./hbars -t Weather -p -w 40 
   ./hbars -t Weather -p -w 40  'temp,33;pressure,14' 
   cat data.txt | ./hbars -W -f 24 -t 'Raspi Data' > data.htm

The code:

#!/bin/bash
#
# hbars.sh - show some text bars
#   pass data as:  label1,value1,unit1,color1; label2,value2,unit2,colour2; ....  
#
width=50
title=""
color=7
pretty=False
web=False
font=24

usage() { 
  echo "usage: hbars [data] [option] "
  echo "  -h --help     print this usage and exit"
  echo "  -c --color    set color to all bars (default 7=white)"
  echo "                 (0-grey,1-red,2=green,3=yellow,4=blue,5=magenta,6=cyan,7=white)"
  echo "  -p --pretty   add different colors to bars (-c overrides)"
  echo "  -t --title    top title"
  echo "  -w --width    max width of bar (default 50)"
  echo "  -W --Web      make output HTML formatted"
  echo "  -f --fontsize fontsize for web output (default 24)"
  echo ""
  echo " examples:"
  echo "   echo 'temp,33,C;pressure,14,psi' | ./hbars -t Weather -p -w 40 "
  echo "   ./hbars -t Weather -p -w 40  'temp,33;pressure,14' "
  echo "   cat data.txt | ./hbars -W -f 24 -t 'Raspi Data' > data.htm"
  echo ""

  exit 0
}
# Show help usage if no pipe data and no cmd line data
if [ -t 0 ]  && [ $# -eq 0 ] ; then
  usage
fi
# Check for command line options
while getopts "hpc:t:w:Wf:" arg; do
  case "$arg" in
    h) usage ;;
    c)  color=$OPTARG ;;
    p)  pretty=True; icolor=0 ;;
    t)  title=$OPTARG ;;
    w)  width=$OPTARG ;;
    W)  web=True;;
    f)  font=$OPTARG ;;
  esac
done
#------------------------------------------------
# Setup formatting for text, Web and color
# -----------------------------------------------
if [[ ${color} != 7 && ${pretty} = True ]]; then
  pretty=False
fi
colidx=0

setcolors() {
if [ $web = True ]; then
  colors=(gray red green yellow blue magenta cyan white)
  titlebold="echo '<h1>'"
  titlereset="echo '</h1>'"
  #color_set='echo "<span style=font-size:$(font)px  >" ' 
  #color_set="printf '<span  style=\"font-size:$(font)px;color:${colors[colidx]}\" >'" 
  color_set="printf '<pre><span style=\"color:${colors[colidx]} ; font-size:${font}px \" >'" 
  color_rs="echo '</span></pre>'"
else
  colors=(0 1 2 3 4 5 6 7 )
  titlebold="tput bold; tput smul"
  titlereset="tput rmul; tput rmso"
  color_set="tput setaf ${colors[colidx]}"
  color_rs="tput rmso"
fi
}
setcolors
#----------------------------------------------
# Get data, check if stdlin, file, if not assume string
#----------------------------------------------
if [ -p /dev/stdin ]; then
        lastarg=$(< /dev/stdin)
else
	lastarg=$(echo "${@: -1}")
	if test -f "$lastarg"; then
	  lastarg=$(<$lastarg)
	fi
fi
# Cleanup the input data
lastarg=$(echo $lastarg | sed 's/\n/;/g; s/  / /g; s/\t//g; s/;;/;/g')
IFS=';' read -r -a array <<< $lastarg

# ensure that there is some data
if [[ ${array} == 0 ]]; then
  echo "No data found"
  exit 0
fi
echo "input:$lastarg"
#exit 0
#------------------------------------
# Get max value and max label length
#------------------------------------
maxval=0
maxlbl=10
#echo "array:${array[@]}"
for element in "${array[@]}"
do 
  IFS=',' read -r -a datapt <<< $element
  if (( $(echo "$maxval < ${datapt[1]}" |bc -l) )); then
	maxval=${datapt[1]}
  fi
  if (( $(echo "$maxlbl < ${#datapt[0]}" |bc -l) )); then
	maxlbl=${#datapt[0]}
  fi
done
#---------------------------------
# Print Title - use bold/underline
#---------------------------------
if [[ ! -z $title ]]; then
  printf "\n %${maxlbl}s " " "
  eval $titlebold
  printf "%s" "${title}" ; printf "\n\n"
  eval $titlereset
fi
#------------------------------------
# Cycle thru data and build bar chart
#------------------------------------
for element in "${array[@]}"
do
# check data values
  IFS=',' read -r -a datapt <<< $element
  # check for empty records
  if [ ${#datapt[0]} = 0 ]; then 
    break
  fi
  label=${datapt[0]}
  if [[ ${label} != "-t*" ]]; then 
	  val=${datapt[1]}
	  sval=$(bc <<< "$width * $val / $maxval")

	  # add color, use 4th item if available
	  if [[ ${#datapt[@]} > 3 && $pretty = False ]]; then
		icolor=${datapt[3]}
	  fi
	  if [[ $pretty = True ]] ; then
		let colidx++
		if [ $colidx -gt 7 ]; then
		  let colidx=1
		fi
	  elif [[ ${#datapt[@]} > 3 ]]; then
		colidx=${datapt[3]}
	  else
	  	colidx=$color
	  fi
	  setcolors
          eval $color_set
          printf " %${maxlbl}s " "$label"
	  bar="printf '█%.0s' {1..$sval}"
	  eval $bar; 
	  # add value and units if available
	  units=""
	  if [[ ${#datapt[@]} > 2 ]]; then
		units=${datapt[2]}
	  fi
	  printf " %d %s\n\n" $val "$units"
	  eval $color_rs 
  fi
done

SQL Output to Bars

With the base code I was able to start doing some more complicated actions, like piping SQL SELECT output. Below is an example from Sqlite3. The first step is to format the SQL output to: label,value;

pete@lubuntu:$ sqlite3 $HOME/dbs/someuser.db "select fname,age from users limit 4"
Brooke|18
Leah|18
Pete|100
Fred|77
pete@lubuntu:$ sqlite3 $HOME/dbs/someuser.db "select fname,age from users limit 4" \
>  | tr  '|' ',' | tr '\n' ';'
Brooke,18;Leah,18;Pete,100;Fred,77;
pete@lubuntu:$ sqlite3 $HOME/dbs/someuser.db "select fname,age from users limit 4" \ > | tr '|' ',' | tr '\n' ';' | ./hbars -t "Sqlite3 Users" -p Sqlite3 Users Brooke █████████ 18 Leah █████████ 18 Pete ██████████████████████████████████████████████████ 100 Fred ██████████████████████████████████████ 77

Web Page with Bars

Support was added for fixed chart width, engineering units and custom line colours. HTML <center> tags were used on the title.

$ cat data.txt
temp,44,degC,1;
humidity,33,%,4;
air pressure,88,mm Hg,5;
rain/precipation,6,mm,6;

$ cat data.txt | ./hbars -W -f 24 -w 50 -t '<center>Raspi Data</center>' > data.htm

Final Comments

Horizontal bars are relatives easy to create in Bash. Unfortunately showing vertical bars and Line charts will require a different approach

TEMPer USB Temperature Sensor

I picked up a TEMPer USB temperature sensor from Walmart for about $12. My goal was to use it with my Home Assistant home automation system.

The model that I picked up supports Windows and it can be used directly in Excel. There is also integration with Home Assistant.

I found that the integration for Linux was tricky. This blog looks at how I got things working with a Raspberry Pi.

USB Connections

The lsusb command can be used to see which USB devices are connected:

pi@pi4:~ $ lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 007: ID 413d:2107  
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

It is important to identify the ID model. The TEMPer USB model that I purchased was 413d:2107.

When I searched the Internet I found that people had created different packages but they were based on specific models. (Check your model).

temper.py

There is a Python library file that can be used on the TEMPer family of sensors. This library component is designed to be run directly as a Python file. It can be accessed by:

pip install temper-py
# Or get the file 
wget  https://raw.githubusercontent.com/ccwienk/temper/master/temper.py

Using the file directly will show the TEMPer device information (Note: use sudo):

$ sudo python3 temper.py
Bus 001 Dev 007 413d:2107 TEMPerGold_V3.1 29.31C 84.76F - - - -

$ sudo python3 temper.py --v
Firmware query: b'0186ff0100000000'
Firmware value: b'54454d506572476f6c645f56332e3120' TEMPerGold_V3.1 
Data value: b'80800b6d4e200000'
Bus 001 Dev 007 413d:2107 TEMPerGold_V3.1 29.25C 84.65F - - - -

$ sudo python3 temper.py --json
[
    {
        "vendorid": 16701,
        "productid": 8455,
        "manufacturer": "",
        "product": "",
        "busnum": 1,
        "devnum": 7,
        "devices": [
            "hidraw0",
            "hidraw1"
        ],
        "port": "1-1.4",
        "firmware": "TEMPerGold_V3.1",
        "hex_firmware": "54454d506572476f6c645f56332e3120",
        "hex_data": "80800b6d4e200000",
        "internal temperature": 29.25
    }
]

I found that the Python app was useful and it could be used on a variety of sensors.

Human Interface Devices

The TEMPer sensor acts like a Human Interface Device. When the TEMPer sensor is connected the dmesg utility can be used to show kernel information:

 $ dmesg -H
[May 1 21:22] usb 1-1.4: USB disconnect, device number 6
[May 1 21:23] usb 1-1.4: new full-speed USB device number 7 using xhci_hcd
[  +0.135008] usb 1-1.4: New USB device found, idVendor=413d, idProduct=2107, bcdDevice= 0.00
[  +0.000020] usb 1-1.4: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[  +0.011366] input: HID 413d:2107 as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:
[  +0.064445] hid-generic 0003:413D:2107.0009: input,hidraw0: USB HID v1.11 Keyboard [HID 413
[  +0.007181] hid-generic 0003:413D:2107.000A: hiddev96,hidraw1: USB HID v1.10 Device [HID 41
(END)

From this output I can see that TEMPer sensor is on the raw human interface device of hidraw1. (For this example the hidraw0 acts like a keyboard but it can’t be used to get temperature info). When I move the TEMPer sensor to a different PC I need to run dmesg again to recheck the hidraw device number, it could be 2,3,4 etc.

A Bash Script

There are some excellent links that got me about 95% of the way. The Bash script below will return a temperature value for my series of TEMPer USB thermometers.

#!/bin/bash
# 
# Get the temperature from a USB Temper Thermometer
#   
#   find the HID device from the kernel msg via dmesg
#   parse the line get HID device
hid=$(dmesg | grep -E  -m 1 'Device.*413d:2107' | \
  awk '{ p=index($4,","); print "/dev/" substr($4,p+1,7) }')
if [ $hid  = "" ]; then
	echo "No TEMPer device found"
else
	#echo "TEMPer device found on: $hid"
	exec 5<> $hid
	# send out query msg
	echo -e '\x00\x01\x80\x33\x01\x00\x00\x00\x00\c' >&5
	# get binary response
	OUT=$(dd count=1 bs=8 <&5 2>/dev/null | xxd -p)
	# characters 5-8 is the temp in hex x1000
	HEX4=${OUT:4:4}
	DVAL=$((16#$HEX4))
	CTEMP=$(bc <<< "scale=2; $DVAL/100")
	echo $CTEMP 
fi

If you are always using the TEMPer in only one place you could simplify the script by hard coding the HID number. I wanted to move the sensor between devices so I added a check find the device.

Control USB Powered Devices

For home automation projects a Raspberry Pi offers a simple low cost approach for controlling a wide variety of devices. Typically these devices are either digitally wiring 0-5Volt devices like motion detectors, or wireless Ethernet devices like smart plugs, but a Raspberry Pi can also control USB powered devices, like USB fan and lights.

In this blog I will look at how to control and monitor Raspberry Pi USB port with two projects. The first project will use Node-Red to create a web dashboard to control USB lights. The second project will turn on USB cooling fans based on the PI’s CPU temperature.

Controlling USB Ports

There are a number of techniques to control USB ports, I found that one of easiest approach is to use the uhubctl utility. To load this utility:

sudo apt-get install libusb-1.0-0-dev
git clone https://github.com/mvp/uhubctl
cd uhubctl
make
sudo make install

The uhubctl utility can be used to view and control USB ports and ports smart USB hubs.

The lsusb command can be used to show connected USB devices

pi@raspberrypi:~ $ lsusb 
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp. SMC9514 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

For the Raspberry Pi 3 and 4 the power on all USB ports is ganged together through port 2, so unfortunately it is not possible to power up/down an individual USB port.

The Pi 4 has two internal USB hubs. Hub 1-1 connects to all the USB port on the USB 2.10 standard. Hub 2 controls to all the ports on the USB 3.00 standard and the Ethernet jack.

The commands to turn on/off/toggle the USB ports and keep the Ethernet jack powered are:

sudo uhubctl -l 1-1 -p 2 -a on
sudo uhubctl -l 1-1 -p 2 -a off
sudo uhubctl -l 1-1 -p 2 -a toggle

The command will return the messages showing the current status, the power requested state and then the new status. To just do an action and remove feedback messages add a 1>&- at the end of the line.

Monitoring USB Power

The uhubctl utility can be used to check if the USB ports are powered. For Raspberry Pi’s the status of Port 2 is the power status for all the ports.

Using some Bash statements the power status can be parsed to just show off or power message. In the next example this Bash statement will be used to show the power status on a Node Red dashboard.

$ sudo uhubctl | grep 'Port 2' | awk '{print $4}'
off

Node Red USB Control Dashboard

Node Red is a visual programming tool that is included with the full desktop install of Raspberry Pi. If Node Red has not been installed see: https://nodered.org/docs/getting-started/raspberrypi

There are number of low cost USB lighting options that can be used with a Raspberry Pi, these include LED strips, eWire and small USB lights. Below is an example of littleBits eWire bit and a USB LED light.

A simple Node Red dashboard can be created that can: 1) turn Raspberry Pi USB ports on and off, and 2) checks the status of power on these ports. The logic would include: 2 dashboard buttons, one dashboard text and 2 exec nodes. The uhubctl utility can be used directly in the exec node.

The first exec node contains the Bash command to turn the USB ports on or off. The “on” or “off” string is sent from the dashboard buttons as a msg.payload this is appended to the command in the exec node. The output from the first exec node triggers the second exec node to get the latest USB port status.

The dashboard text node is configured to show the power status text as a large uppercase heading.

Once the logic is complete, the Deploy button on the right side of the menu bar will make the dashboard available to web clients at: https://raspberry_pi_address:1880/ui .

For my example I added an enhancement to include a countdown or sleep timer.

Raspberry Pi Cooling Fan

Raspberry Pi’s have a number of different cooling options. For this example I used a two littleBits fans that I placed on a littleBits mounting plate.

The vcgencmd measure_temp command will return the Pi CPU temperature. By adding some awk script it’s possible to parse out just the float value:

$ vcgencmd measure_temp 
temp=46.2'C
$ vcgencmd measure_temp | awk '{ print substr($1,6,4)}'
45.6

The bc (arbitrary precision calculator) command can be used with the math library (-l option) to check if one float is greater than another float number:

$ echo "33.4 > 36.1" | bc -l
0
$ echo "38.4 > 36.1" | bc -l
1

A simple script to loop through and check the temperature against a limit, and then turn on/off a fan would be:

#!/bin/bash
#
# Check the Pi temperature and turn on fan if too high
# 
tlim="46.0"
while :;
do
  tnow=$(vcgencmd measure_temp | awk '{ print substr($1,6,4)}')
  # check the CPU temp vs. the limit
  if (( $(echo "$tnow > $tlim" | bc -l ) )) ; then
     # CPU temp is above limit, turn on fan
     sudo uhubctl -l 1-1 -p 2 -a on 1>&-
  else
     # CPU temp is below limit, turn off fan
     sudo uhubctl -l 1-1 -p 2 -a off 1>&-
  fi
  sleep 10
done

This code can be enhanced to show when the fan needs to be turned on and off:

#!/bin/bash
#
# Check the Pi temperature and turn on fan if too high
#

# Start with Fan on
tlim="46.0"
fan="on"
sudo uhubctl -l 1-1 -p 2 -a $fan 1>&-

# Periodically check CPU temperature
while :;
do
  tnow=$(vcgencmd measure_temp | awk '{ print substr($1,6,4)}')

  if (( $(echo "$tnow > $tlim" | bc -l ) )) ; then
      if [ $fan = "off" ] ; then
    	fan="on"
	    echo "$(date +'%T')  Temp: $tnow - Turn fan $fan"
        sudo uhubctl -l 1-1 -p 2 -a $fan 1>&-
      fi
  else
      if [ $fan = "on" ] ; then
    	fan="off"
	    echo "$(date +'%T')  Temp: $tnow - Turn fan $fan"
        sudo uhubctl -l 1-1 -p 2 -a $fan 1>&-
      fi
  fi
  sleep 10
done

Power Control on other Controllers

A Raspberry Pi can control the power on other controllers, below is an example of a Pi 4 powering an Arduino UNO, an Arduino Nano (clone) and a BBC Micro:bit controller.

For external modules that don’t support Wifi or real time clocks a Raspberry Pi could be used as an easy way to power up/power down these external controllers.

It’s important to realize that a Raspberry Pi is not designed to power devices that have a high power requirement.

The Raspberry Pi 3 and 4 can have a maximum USB port output of 1200mA for all 4 ports combined, with no per-port limits (meaning, all 1200mA is available on a single port if no others are in use). This 1200mA limit assumes that the power is from a 2.5A power supply for the Pi3 or the 3A power supply for the Pi4.

If you connecting smart USB devices like devices like memory sticks or 3rd party controller, the device manufacture has a defined MaxPower rating that can be found once the device is connected.

The command lsusb -v will give a very long list of the full vendor information for all the connected devices. To get just the MaxPower for each device on the Raspberry Pi USB internal bus enter:

sudo lsusb -v  2>&- | grep -E  'Bus 00|MaxPower'

When this command is run with an Arduino Nano, Arduino UNO and a BBC Micro:bit and a memory stick, the total power requirements can be seen.

pi@pi4:~ $ sudo lsusb -v  2>&- | grep -E  'Bus 00|MaxPower'
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    MaxPower                0mA
Bus 001 Device 004: ID 1a86:7523 QinHeng Electronics HL-340 USB-Serial adapter
    MaxPower               96mA
Bus 001 Device 003: ID 2341:0043 Arduino SA Uno R3 (CDC ACM)
    MaxPower              100mA
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
    MaxPower              100mA
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    MaxPower                0mA

With this example the total used USB power is 296mA (0 +96+100+100+0), and this is within the Raspberry Pi specs.

A Bash command to calculate the total required bus power would be:

pi@pi4:~ $ (sudo lsusb -v 2>&- | grep MaxPower | grep -o -E '[0-9]+' ) | awk '{ sum += $1} END {print "\nTotal= " sum " mA"}' 

Total= 296 mA

Unfortunately devices like USB lights use the USB connection for power only so they do not appear in lsusb, so you’ll need to check the manufacturers literature for power requirements.

Final Comments

I prefer using direct wired GPIO pin connections or Wifi devices over using USB powered devices, however it’s nice to know that you have the USB option if you need it.

For kids projects that use littleBits or Micro:bits using a Pi with Node Red offers a nice way to remotely control them.

Automate Web Page Logins

To ensure a safe number of users during Covid we had to log into Web sites for access to activities like pools, gyms and ski hills. These precautions made sense, however the booking process was awkward and was easy to miss an activity if you weren’t signed up early enough.

Luckily there are some great Linux tools that can be used to automate web logins. In this blog I’ll look at two techniques:

  • Keyboard simulation using xte and xdotool
  • Python accessing the HTML on page using Selenium

Linux Keyboard Simulation

If you are working in MS-Windows take a look at SendKeys command.

In the Linux environment there a number of choices, such as xautomation and xdotool.

The xdotools package is feature rich with special functions for desktop and windows functions. The xautomation package was a little simpler and it focuses on keyboard and mouse simulation.

I found that it was very useful to install wmctrl, this allows you to easily find out which windows are running and it can set the active window with a substring (you don’t need the full window name).

To install xautomation and wmctrl in Ubuntu:

sudo apt-get install xautomation wmctrl

To install xdotool in Ubuntu:

sudo apt-get install xdotool

Log in Example with xte

To create an automation script you need to do the required steps once manually and document what is a Tab, an entry field or a Return key. Some trial and error work will probably be required for timing if you are working between pages links or with pages that have a slow call up time.

A good simple example is to try and log into Netflix.

The following Bash script will : 1) open a Chrome browser page, 2) set the focus to the page and 3) sent the correct tab, text and a return key.

#!/bin/bash
# netflix_login.sh - script logs into Netflix
#

url="https://www.netflix.com/ca/login"
email="my_email.com"
pwd="my_password"

chromium-browser $url & #open browser to

# wait for the page to open, the set focus to it
sleep 2
wmctrl -a "Netflix - Chromium"
sleep 1 # allow time to get focus before sending keys
xte "key Tab"
xte "key Tab"
xte "str $email"
xte "key Tab"
xte "str $pwd"
xte "key Tab"
xte "key Tab"
xte "key Return"

echo "Netflix Login Done..."

Use wmctrl -l to see which windows are open.

A specific window can have the focus based on a substring, with the -a options

The xte command uses a string to pass key, text and mouse actions to the active window.

Park Booking using xdotool

The xdotool syntax to send keystrokes and text is very similar to xte, but with a few extra features.

The park booking example is a little bit more complex because a booking time needs to be selected from a list. It might be possible to tab to the required time, but a safer way would be to do a search to find the required time.

Neither the xte nor the xdotool utilities support a search text function. A simple workaround is to use the web browser’s search function. To have keystrokes go to the result of the browser search it’s important to enable Caret navigation.

The Caret dialog is shown by entering F7. It’s important to note that the Caret Enable/Cancel or Yes/No buttons will vary between different browsers.

For the park booking page, the animation script needs to manage 8 entry fields. To keep things simple I’ll pass the date in the URL.

The important issue is to book a time slot, for this Control-F will call up the Search Dialog. After the required time is entered, the navigation moves to the selected time. The next step is to close the Search Dialog, this done by sending three tabs and a return key.

The final bash script is shown below. One of the useful features of xdotool is that it can do repeat key strokes with a delay between entries.

#!/bin/bash
# book10am.sh - make a 10:00 park booking
#
sdate="startDate=2021-04-23" #adjust the date
url="https://book.parkpassproject.com/book?inventoryGroup=1554186518&&inventory=1229284276&$sdate"

chromium-browser $url & #open browser to park booking page
sleep 5 # wait for browser to come up
wmctrl -a "Chromium"
sleep 2
# Turn on caret browsing
xdotool key F7
xdotool key Return
sleep 1

# tab to 'Time Slot' area 
tabcnt=8
xdotool key --repeat $tabcnt --delay 100 Tab

xdotool key Return
sleep 1

# Search for 10:00 time and select it
xdotool key ctrl+f 
xdotool type '10:00'
xdotool key Return
# Close find dialog and select time
xdotool key Tab Tab Tab Return Return

echo "Park Time Booking Complete"

Script Limitations with xdotool and xte

Using xdotool or xte is great for simple web page automation where the HTML form items are sequential and no special decision making is required.

Unfortunately I found that when I tried to book a park time on the weekend I started to see some limitation. During busy times if I tried to book by time the xte or xdotool could not determine if the time slot was full.

A simple workaround would be to search for the first ‘Available’ or ‘Not Busy’ slot but this doesn’t allow you to pick times that you like.

For projects that require some logic (like picking a good time from a list of time), Selenium with Python is an excellent fit.

For more complex web automation projects Selenium with Python is an excellent fit.

Installing Selenium with Python

Selenium is a portable framework for testing web applications, with server/client tools and IDE’s.

The Selenium WebDriver component sends commands from client APIs directly to a browser. There are webdriver components for Firefox, Google Chrome, Internet Explorer, Safari, Opera and Edge. Client API’s are available for C#, GO, Java, JavaScript, PhP, Python and Ruby.

For details on extra information on installation of the different webdrivers see: https://www.selenium.dev/downloads/

To install the Linux 32-bit Selenium Driver (geckodriver) for Firefox:

wget https://github.com/mozilla/geckodriver/releases/download/v0.29.1/geckodriver-v0.29.1-linux32.tar.gz
tar -xvzf geckodriver-v0.24.0-linux32.tar.gz
chmod +x geckodriver
sudo mv geckodriver /usr/local/bin

To install the Selenium library for Python:

pip install selenium

The big difference between using xte and selenium is that selenium can directly access the HTML code of the selected web page. The xte approach to accessing form items is to tab to them like you would with a keyboard. Selenium allows code to access an HTML item using its id name.

Selenium Log In Example

Like using xte the user needs to do some manual work before the script is written.

Once the required web page is open, the Web Developer Inspector tool can used to examine HTML code. To access the Inspector, Select Tools > Web Developer > Inspector from the top menu bar, or use the shortcut control-shift-C.

For the Netflix log in example the key HTML items are the “email or phone number” input and the the password input. Using the Inspector these id’s (or names) can be found.

The Python code to log into Netflix is:

#
# netflix_login.py - automate Netflix Login
#
from selenium import webdriver

url="https://www.netflix.com/ca/login"
email="my_email.com"
pwd="my_password"

browser = webdriver.Firefox()

browser.get(url)

# wait for page to refresh
browser.implicitly_wait(10) 

username = browser.find_element_by_id('id_userLoginId')
username.send_keys(email)

password = browser.find_element_by_id('id_password')
password.send_keys(pwd)

password.submit()

print("Login Complete")

When a web page is called it’s important to give the page some time to refresh. The implicity_wait(10) call will wait up to 10 seconds for a Selenium query. HTML Items can be found by either find_element_by_id() or by find_element_by_name(). The send_keys() method is used to pass text strings to <input> tags. Finally calling submit() will send all the form data to the requested action.

Using Selenium Searches

From the earlier park booking example we saw that xte had some limitations when a variable lists of options were presented. Luckily Selenium has a number of functions that can be used for searching HTML tags and text.

The first step is to manually open the web page and inspect the structure.

For this example the Inspector shows that each status entry in the list has a <div class=”jss97″>. with the Available items having a <div class=’jss100′> and the “Not Busy” items being a <div class=’jss100′>. The very top level is <div class=”jss94″>, this has both the times and the status messages.

Now that I know how the park times are defined I can check for the first “Not Busy”entry, or I can check a specific time and see if it’s busy or not.

Below is a code example that shows the first “Not Busy” time. To select a specific time/status entry use the click() method (itimes.click()).

Final Comments

Using simulated keyboard and mouse movements is a nice tool for automating simple web pages.

If you need to look at more complex web automation solutions check out Selenium, it has a Python library that allows you to access the DOM object of a web page.

Orange Pi – $2-$50 Raspberry Pi Competitor

Orange Pi is a low-cost Raspberry Pi competitor. It is developed by the Chinese Shenzhen Xunlong CO Software company. Just like the Raspberry Pi, Orange Pi is an Open Source project.

Orange Pi offers a wide variety of offerings, that start at $2 and go up from there.

I purchased an Orange Pi Lite for about $15, and I’ve been quite happy with it.

Compared to the Raspberry Pi, Orange Pi is:

  • less expensive. Faster for the price
  • has more hardware options and form factors

This blog documents some of my installations steps to get an Orange Pi (Lite) running with Raspberry Pi functionality.

Some Interesting OrangePi Offers

There are quite a few different modules that are available. Some of the ones that I found interesting are:

Orange Pi Zero – there a few model, the 256MB LTS ($2), to the Zero 2 ($30).

Orange Pi 3G-IOT-B – includes a built-in 3G SIM card support ($22). There are also 4G models.

Base Installation

Like the Rasp Pi there are a number of OS options that can be installed. I chose the Armbian OS, which seems to be the most popular.

There are a number of ways to put an image on an SD chip, I like: pi imager. Once an image has been installed, the command line can be used to get the basic things setup and installed

  ___  ____  _   _     _ _       
 / _ \|  _ \(_) | |   (_) |_ ___ 
| | | | |_) | | | |   | | __/ _ \
| |_| |  __/| | | |___| | ||  __/
 \___/|_|   |_| |_____|_|\__\___|
                                 
Welcome to Armbian buster with Linux 5.4.43-sunxi

System load:   0.00 0.00 0.00  	Up time:       21 min		
Memory usage:  41 % of 492MB  	IP:            192.168.0.113
CPU temp:      52°C           	
Usage of /:    16% of 15G    	

Last login: Sat Apr 17 12:28:07 2021 from 192.168.0.111

On Pi the config tool is called raspi-config, on the Ambian OS it’s : armbian-config

This tool is used to setup initial key things like: Wifi, Bluetooth, SSH, Desktop setting etc.

Depending on the OS that has been installed you may have everything you need or you may need to do further installs. For myself I needed to load “Pi type” features such as: the gpio utility, Python GPIO library and Node-Red.

I logged in as root for all my base installations.

GPIO (WiringPi) Utility

The gpio utility is super useful for manual seeing and setting GPIO pins.

To install gpio :

git clone https://github.com/orangepi-xunlong/WiringOP
cd WiringOP
chmod +x ./build
sudo ./build

The pinouts are different that the Rasp Pi, to see them enter: gpio readall

root@orangepilite:~# gpio readall
 +------+-----+----------+------+---+OrangePiH3+---+------+----------+-----+------+
 | GPIO | wPi |   Name   | Mode | V | Physical | V | Mode | Name     | wPi | GPIO |
 +------+-----+----------+------+---+----++----+---+------+----------+-----+------+
 |      |     |     3.3V |      |   |  1 || 2  |   |      | 5V       |     |      |
 |   12 |   0 |    SDA.0 |  OFF | 0 |  3 || 4  |   |      | 5V       |     |      |
 |   11 |   1 |    SCL.0 |  OFF | 0 |  5 || 6  |   |      | GND      |     |      |
 |    6 |   2 |      PA6 |  OFF | 0 |  7 || 8  | 0 | OFF  | TXD.3    | 3   | 13   |
 |      |     |      GND |      |   |  9 || 10 | 0 | OFF  | RXD.3    | 4   | 14   |
 |    1 |   5 |    RXD.2 |  OFF | 0 | 11 || 12 | 0 | OFF  | PD14     | 6   | 110  |
 |    0 |   7 |    TXD.2 |  OFF | 0 | 13 || 14 |   |      | GND      |     |      |
 |    3 |   8 |    CTS.2 |  OFF | 0 | 15 || 16 | 0 | OFF  | PC04     | 9   | 68   |
 |      |     |     3.3V |      |   | 17 || 18 | 0 | OFF  | PC07     | 10  | 71   |
 |   64 |  11 |   MOSI.0 |  OFF | 0 | 19 || 20 |   |      | GND      |     |      |
 |   65 |  12 |   MISO.0 |  OFF | 0 | 21 || 22 | 0 | OFF  | RTS.2    | 13  | 2    |
 |   66 |  14 |   SCLK.0 |  OFF | 0 | 23 || 24 | 0 | OFF  | CE.0     | 15  | 67   |
 |      |     |      GND |      |   | 25 || 26 | 0 | OFF  | PA21     | 16  | 21   |
 |   19 |  17 |    SDA.1 |  OFF | 0 | 27 || 28 | 0 | OFF  | SCL.1    | 18  | 18   |
 |    7 |  19 |     PA07 |  OFF | 0 | 29 || 30 |   |      | GND      |     |      |
 |    8 |  20 |     PA08 |  OFF | 0 | 31 || 32 | 0 | OFF  | RTS.1    | 21  | 200  |
 |    9 |  22 |     PA09 |  OFF | 0 | 33 || 34 |   |      | GND      |     |      |
 |   10 |  23 |     PA10 |  OFF | 0 | 35 || 36 | 0 | OFF  | CTS.1    | 24  | 201  |
 |   20 |  25 |     PA20 |  OFF | 0 | 37 || 38 | 0 | OFF  | TXD.1    | 26  | 198  |
 |      |     |      GND |      |   | 39 || 40 | 0 | OFF  | RXD.1    | 27  | 199  |
 +------+-----+----------+------+---+----++----+---+------+----------+-----+------+
 | GPIO | wPi |   Name   | Mode | V | Physical | V | Mode | Name     | wPi | GPIO |
 +------+-----+----------+------+---+OrangePiH3+---+------+----------+-----+------+

Python GPIO Library

The Raspberry Pi Python GPIO library as been ported to the Orange Pi and it’s called OPi.GPIO. To install it:

# if required install PIP for Python 2 and 3
apt install python-pip python3-pip

# install Orange Pi GPIO library for Python 2 and 3
pip install OPi.GPIO
pip3 install OPi.GPIO

A quick test from to test that things are working:

root@orangepilite:~# python3
Python 3.7.3 (default, Dec 20 2019, 18:57:59) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import OPi.GPIO as GPIO
>>> GPIO.setmode(GPIO.BOARD)
>>> GPIO.setup(12, GPIO.OUT)
>>> # set pin 12
... 
>>> GPIO.output(12, 1)
>>> GPIO.input(12)
1
>>> GPIO.cleanup()

Node-Red

To install Node-Red, (logged in as root), use the following command, and follow the prompts (don’t install Raspberry support). This install takes about 20 minutes and Node.js will also be added:

bash <(curl -sL https://raw.githubusercontent.com/node-red/linux-installers/master/deb/update-nodejs-and-nodered)

The next step is to install the Orange PI GPIO support:

cd $HOME/.node-red
npm install node-red-contrib-opi-gpio

To manual start Node-Red enter: node-red-start & . To manually stop Node-Red: node-red-stop. See the Node-Red documentation for more info.

A simple circuit to set and read GPIO would be:

Final Comments

A Orange Pi is awesome for generic or simple hardware projects.

If however you’re doing projects where you need tops, for motor controls or relays then I would still to a Raspberry Pi.

Re-purpose an eReader

My daughter and I looked at trying to re-purpose an old eReader. Our goal was to make a kitchen kiosk display that would show us items of daily interest. For us that included news, stocks and weather data that we pulled from the Internet, along with some local data from Arduino, Raspberry Pi and a Home Assistant nodes.

In this blog we’ll look at:

  • How to install a different OS on the eReader
  • Using Linux on an eReader
  • Python eReader considerations
  • Kiosk mode Web Browser pages

Opening up the eReader

For our project we used a Kobo mini, so the procedure to load a new OS will vary somewhat based on the eReader manufacturer.

Once you open up the eReader, you’ll have access to an microSD inside the unit.

I would highly recommend keeping the original SD in case that you ever need to roll back to the original setup.

For eReaders there are two main OS choices, Android or Linux. For our project we felt that Linux would be a cleaner option. Images can be downloaded from: https://www.dropbox.com/sh/snsdg1c5cg21kws/3LfelXgbGe.

If you are trying to re-purpose other brands of eReaders there appears to be lots of how to guides, for example for Kindle see: https://www.lifehacker.com.au/2016/07/how-to-jailbreak-your-kindle .

Installing Debian Linux

Once you’ve downloaded your required OS you’ll need to put in on a micro-SD chip.

There are a number of different tools to move and copy images, because I’m a Raspberry Pi user I like to use the Raspberry Pi Imager (rpi-imager). The rpi-imager utility runs on Linux, Windows and Mac OS. To install it on Linux:

sudo snap install rpi-imager

Using rpi-imager, select the “use custom” option.

Depending on your eReader and the OS you might be ready to go. Unfortunately for our Kobo mini installation we needed to another step. The added step that we needed to do moved some files from the original SD chip on to our new SD chip. This added step gave us a clean double-boot install so we could either run the original Kobo software or boot into Debian Linux. Our added steps were:

# insert the original microsd
sudo dd if=/dev/mmcblk0 bs=512 skip=1024 count=1 of=/home/you/path/to/image/original.img
# insert the new microsd with the debian image on it
sudo dd if=/home/ian/Desktop/kobo/original.img bs=512 seek=1024 count=1 of=/dev/mmcblk0

To setup the Wifi we needed to boot into the Kobo system and configure our network settings. Once that was complete we were able to have Wifi networking on the Linux side.

Linux on the eReader

The first step is to enable the Wifi, this option should be on the main menus.

eReaders have great battery life when they are used as an eReader, however when they are 100% on Wifi their battery life will be more like a standard tablet, with 5-6 hours of life.

The eReader has a floating keyboard so you can do some work directly on the unit, but doing an ssh connection from a PC is definitely easier.

The menuing on the eReader will vary based on the OS that is loaded. On our Kobo, the Debian OS used the awesome window manager. To add items into the menus look for the file: .config/awesome/rc.lua . We added two extra entries, one for a Python app and the second for a Browser kiosk app :

-- This is in the /awesome/rc.lua menu file
-- ...

menuapps = {
   { "MyApp","/my_path/g4.py"},
   { "MyWebpage","/my_path/mywebpage.sh"},
   { "Firefox", "firefox" },
   { "FBReader", "fbreader" },
   { "Calculator", "xcalc" }
}

Python on an eReader

For our first Python test app we wanted to :

  • 100% fill the eReader screen
  • Find different Font sizes that worked well on an eReader
  • Have a large “Close” button.
# Simple Tkinter Clock
#
from Tkinter import *
import time

def update_clock():
        now = time.strftime("%H:%M")
        lbl_time.configure(text=now)
        today = time.strftime("%A %B %d")
        lbl_date.configure(text=today)
        window.update()
        window.after(5000, update_clock)

window=Tk()

lbl_time=Label(window, text="", font=("Helvetica", 128))
lbl_time.pack()
lbl_date=Label(window, text="", font=("Helvetica", 64))
lbl_date.pack()
btn=Button(window, text="Close",font=("Helvetica", 32),bg='grey', command=window.destroy)
btn.pack()

window.title('Kitchen Kiosk')
window.geometry("800x600+0+0")
window.attributes("-fullscreen", True)
window.after(1000, update_clock)
window.mainloop()

We found that it took a bit of time to play with font sizing and gray tones before we had something that we liked. Our final Python app used weather data from our Home Assistant Node and we built some custom gauges.

Kiosk Browser App

Our second app showed a custom web page that collected data from a Raspberry Pi and our Home Assistant node.

The plan was to show the data on the web browser in full screen or kiosk mode on the eReader.

To run Firefox in kiosk mode:

firefox --kiosk http://mysite/thepage.htm

On our eReader the browser was iceweasel, which is a lighter weight browser which unfortunately does not support kiosk mode. A workaround for browsers that don’t support kiosk mode is to use xdotool which allows you to simulate mouse and keyboard actions. We wrote a script that opened iceweasel and then re-positioned the window:

#!/bin/bash
# mywebpage.sh - open web browers to our page and go full screen

iceweasel http://192.168.0.111:8080/ &
sleep 15
xdotool key F11

Final Comments

The Debian OS that we loaded on our eReader was fairly old and it didn’t support Python 3.7 or Firefox, but we found alternatives and we still could make it work.

If your applications doesn’t need to be constantly updated, for example only check weather and stocks every 15 minutes or so, then you could turn on the Wifi, get the data, then turn off the Wifi. This could greatly increase the battery life on the eReader.

Home Assistant (REST) API

There are a few methods to communicate with Home Assistant. In this blog I wanted to document my notes on using the REST API.

Getting Started

I found that loading the File Editor Add-on made configuration changes quite easy. To load the File Editor, select the Supervisor item, then Add-on Store:

With the File Editor option you be able to modify your /config/configuration.yaml file. For this you’ll need to add an api: statement. I also added a command line sensor that shows the Raspberry Pi CPU idle time. I did this so that I could see a dynamic analog value:

After I made these changes I restarted my HA application, by the “Configuration” -> “Server Controls”.

Next I needed to create a user token. Select your user and then click on “Create Token” in the Long-Lived Access Tokens section. This token is very long and it only is shown once, so copy and paste it somewhere save.

Access the REST API with CURL

Curl is a command line utility that exists on Linux, Mac OS and Windows. I work in Linux mostly and it’s pre-installed with Ubuntu. If you’re working in Windows you’ll need to install CURL.

Getting started with curl isn’t required and you got straight to programming in your favourite language, however I found that it was usefully testing things out in curl before I did any programming.

The REST API is essentially an HTTP URL with some headers and parameters passed to it. For a full definition see the HA API document. The key items in REST API are:

  • Request type – GET or POST (note: there are other types)
  • Authorization – this is where the user token is passed
  • Data – is used for setting and defining tags
  • URL – the Home Assistant URL and the end point (option to view or set)

To check that the HA API is running a curl GET command can used with the endpoint of /api/.

$ curl -X GET -H "Authorization: Bearer eyJ0eXAiO....zLjc"   http://192.168.0.103:8123/api/

{"message": "API running."}

The user token is super long so your can use the \ character to break up your command. For example:

curl -X GET \
   -H "Authorization: Bearer eyJ0eXAiOiJKV......zLjc" \
   http://192.168.0.106:8123/api/states

Read a Specific HA Item

To get a specific HA item you’ll need to know its entity id. This can found by looking at the “Configuration” -> “Entities” page:

For my example I created a sensor called Idle Time, its entity id is: sensor.idle_time.

A curl GET command with the endpoint of /states/sensor.idle_time will return information on this sensor. The full curl command and the results would look like:

$ curl -X GET   -H "Authorization: Bearer eyJ0eXAiOiJKV1Q....zLjc"  \   http://192.168.0.103:8123/api/states/sensor.idle_time

{"entity_id": "sensor.idle_time", "state": "98.45", "attributes": {"unit_of_measurement": "%", "friendly_name": "Idle Time"}, "last_changed": "2020-12-12T17:34:10.472304+00:00", "last_updated": "2020-12-12T17:34:10.472304+00:00", "context": {"id": "351548f602f5a3887ff09f26903712bc", "parent_id": null, "user_id": null}}

Write to a New HA Item

A new or dynamic items can be created and written to remotely using a POST command with the definitions included in the data section. An example to create an entity called myput1 with a value of 88.6 would be:

curl -X POST \
   -H "Authorization: Bearer eyJ0eXAiOi....zLjc" \
   -H "Content-Type: application/json" \
   -d '{"state":"88.6", "attributes": {"unit_of_measurement": "%", "friendly_name": "Remote Input 1"}}' \
   http://192.168.0.103:8123/api/states/sensor.myinput1

This new entity is now available to HA and shown on the dashboard.

Write to a Switch

If you have a writeable device such as a switch you can use the REST to remotely control it.

For myself I have a Wemo switch with an entity name of : switch.switch1.

To control the switch the entity id is passed in the data section and the endpoint uses either a turn_on or turn_off parameter.

curl -X POST \
   -H "Authorization: Bearer eyJ0eXAiOiJ....zLjc" \
   -H "Content-Type: application/json" \
   -d '{"entity_id": "switch.switch1"}' \
   http://192.168.0.103:8123/api/services/switch/turn_on

Python and the HA API

Python can parse the JSON responses from the reading a sensor value:

from requests import get
import json

url = "http://192.168.0.103:8123/api/states/sensor.idle_time"
token = "eyJ0eXAiOiJK...zLjc"

headers = {
    "Authorization": "Bearer " + token,
    "content-type": "application/json",
}

response = get(url, headers=headers)

print("Rest API Response\n")
print(response.text)

# Create a json variable
jdata = json.loads(response.text)

print("\nJSON values\n")
for i in jdata:
    print(i, jdata[i])

The output will be something like:

Rest API Response

 {"entity_id": "sensor.idle_time", "state": "98.46", "attributes": {"unit_of_measurement": "%", "friendly_name": "Idle Time"}, "last_changed": "2020-12-12T19:29:10.655530+00:00", "last_updated": "2020-12-12T19:29:10.655530+00:00", "context": {"id": "2509c01cadb9e5b0681fa22d914e7b10", "parent_id": null, "user_id": null}}

 JSON values

 entity_id sensor.idle_time
 state 98.46
 attributes {'unit_of_measurement': '%', 'friendly_name': 'Idle Time'}
 last_changed 2020-12-12T19:29:10.655530+00:00
 last_updated 2020-12-12T19:29:10.655530+00:00
 context {'id': '2509c01cadb9e5b0681fa22d914e7b10', 'parent_id': None, 'user_id': None}

To write a value to myinput1 in Home Assistant:

from requests import post
import json

url = "http://192.168.0.103:8123/api/states/sensor.myinput1"
token = "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJkMDI2YjAxY2VkZWU0M2E1OWY1NmI1OTM2OGU1NmI0OSIsImlhdCI6MTYwNzc5Mzc0NCwiZXhwIjoxOTIzMTUzNzQ0fQ.qEKVKdadxNWp249H3s_nmKyzQMIu5WDQkS9hiT-zLjc"

mydata = '{"state":"99.3", "attributes": {"unit_of_measurement": "%", "friendly_name": "Remote Input 1"}}'

headers = {
    "Authorization": "Bearer " + token,
    "content-type": "application/json",
}

response = post(url, headers=headers,data =mydata)

print("Rest API Response\n")
print(response.text)

# Create a json variable
jdata = json.loads(response.text)


print("\nJSON values\n")
for i in jdata:
    print(i, jdata[i])

The output will look something like:

Rest API Response

 {"entity_id": "sensor.myinput1", "state": "99.3", "attributes": {"unit_of_measurement": "%", "friendly_name": "Remote Input 1"}, "last_changed": "2020-12-12T20:40:05.797256+00:00", "last_updated": "2020-12-12T20:40:05.797256+00:00", "context": {"id": "31b422d02db41cde94470ebae7fac48c", "parent_id": null, "user_id": "1392a10c7bbb4cf0891a7f8a351740c7"}}

 JSON values

 entity_id sensor.myinput1
 state 99.3
 attributes {'unit_of_measurement': '%', 'friendly_name': 'Remote Input 1'}
 last_changed 2020-12-12T20:40:05.797256+00:00
 last_updated 2020-12-12T20:40:05.797256+00:00
 context {'id': '31b422d02db41cde94470ebae7fac48c', 'parent_id': None, 'user_id': '1392a10c7bbb4cf0891a7f8a351740c7'}

Final Comments

A REST API interface allows foreign devices such as PCs, Raspberry Pi and Arduino module to be used as remote I/O devices.

There are REST client HTTP libraries that are available for the Arduino, however it might be cleaner to implement an MQTT interface instead.

Make your Python Apps Run Faster

Python apps on lower end hardware like a Raspberry Pi can be a bit slow but luckily there are some options that you can do to improve things.

There are a number of interesting packages that allow you to compile, interpret or repackage your Python apps. In this blog I’d like to highlight two packages that I have had good success with:

  • Pypy – a replacement to the native Python interpreter. Your code can run more that 4 times faster !
  • Nuitka – a native Python utility to compile Python apps to C code !

Pypy – a faster Python

The Pypy site has some performance results, and they state that on average Pypy will run 4.2 times faster than native Python, for my test run Pypy ran 9 times faster than native Python.

The improved performance of Pypy is due to its just-in-time compiler, as opposed to the native Python’s line-by-line interpreter.

Guido van Rossum, creator of Python, has been even been quoted saying: “If you want your code to run faster, you should probably just use PyPy.”

Pypy has a Python 2.7 and 3.6 version that is available for Linux, MacOS and Microsoft Windows. To install the Pypy 3 version in Ubuntu :

sudo add-apt-repository ppa:pypy/ppa
sudo apt update
sudo apt install pypy3

Pypy3 can also be installed using snap, (this might be the easiest way on a Raspberry Pi):

#if you need to install snap 
sudo apt install snapd 
# reboot after snap is installed 
sudo snap install pypy3 --classic

The nice thing about Pypy is that you can use your base Python code as is, and you can do basic testing with Pypy in command line mode.

To run your Python application from the command line substitute python with pypy (or pypy3).

$ pypy3 myapp.py

If you running your Python app as an executable script (i.e. with the chmod +x myapp) then change the first line of your script from #!/usr/bin/python to : #!/usr/bin/pypy3

Pypy Libraries and Limitations

Pypy has an excellent selection of supported libraries , but it is important to note that not all Python libraries are supported. It’s important to check to see if your required library in supported. Unfortunately TKinter is not in the supported list.

To load a Python library into Pypy3, the Python package installer (pip) module needs to be installed:

$ wget https://bootstrap.pypa.io/get-pip.py
$ pypy3 get-pip.py

Once Pypy3 has pip installed, Python packages can be loaded:

$ pypy3 -m pip install some-pymodule

I found that I was able to load some of the “lighter” modules such as: bottle, requests, beautifulsoup4, and pika etc. without any issues. However some of the “heavier” modules such as Numpy and MatPlotLib would not load directly. If you’re planning on using some of the “heavier” Python modules it is recommended that Pypy be run in virtualenv.

Nuitka – a Python compiler written in Python

Nuitka is the Python compiler. It is written in Python. It is a seamless replacement or extension to the Python interpreter and compiles every version of Python (2.6, 2.7, 3.3, 3.4, 3.5, 3.6, 3.7, and 3.8).

Nuitka has two requirements: 1) a “C” compiler, and 2) Python must be installed on the target machine.

Nuitka works on Linux, MacOS X and Window. For Linux the gcc compiler is the default (5.1 or later). The clang compiler is used on macOS X, and for Windows MinGW64 or Visual Studio 2019 compilers can be used.

To install Nuitka:

python -m pip install nuitka

To compile a test project (test1.py) simply enter:

python -m nuitka test1.py

The Nuitka compiled program will be test1.exe in Windows and test1.bin in Linux.

Performance Testing

Performance testing is very subjective and the results can vary greatly based on so many factors. To have some kind of standard I tried using Python’s pystone benchmark testing utility. I used a Raspberry Pi 3 with a number of cases.

The testing showed some interesting results.

  • The jython (Java VMS Python) interpreter was over 2 times slower than native Python
  • iPython (used in Jupyter notebooks) was almost the same speed at native Python
  • PyInstall (a cool Python packager) was 33% slower than native Python
  • A Nuitka executable was 30% faster than native Python
  • Pypy is 9 times faster than native Python and 6.5 times faster than Nuitka

Test Results with pystone.py

TestSpeedFactor
Jython 50000 passes = 5.406222.13
Python3 50000 passes = 2.53779s1
iPython 50000 passes = 2.547941
PyInstall package50000 passes = 3.38577s1.33
Nuitka Executable 50000 passes = 1.78449s0.70
Pypy3 50000 passes = 0.272579s0.11

Summary

There are a number of other choices that can be used to make Python code run faster, (such as Pyston and Cython) but I found Pypy and Nuitka to be the best supported.

Pypy is incredibly fast, and if you’re using “lighter” weight Python modules its a fantastic fit. I think that Pypy would be great for custom microWeb server and SQLite apps. However for data mining and AI projects you might have some issues getting Pypy working with these “heavier” Python modules.

I was super impressed with Nuitka it compiled a variety of my projects without any issues. It had a nice speed improvement. Nuitka also had no problem with GUI libraries like Tkinter, PySimpleGUI and tk_tools or data management libraries like Numpy or Pandas. However Nuitka isn’t bulletproof and I had some problems with video libraries (cw2).