Bash with MQTT

I’m working on re-purposing an old router. One of my goals is to bring sensor and performance data from the router to a Home Assistant node. The router doesn’t not have a lot of space so I’d prefer to use Bash rather than Python for MQTT communications.

While I was working on the project I wanted to use some simple tools to view the data, unfortunately I wasn’t able to find any information on how to make a good MQTT client in Bash.

This blog documents how I used Bash to show bar charts of MQTT data.

Setup

The first step is to install the Mosquitto command line tools on the OpenWrt router:

# Update the package manager list
opkg update
# Install the MQTT client utiliy
opkg install mosquitto-client-nossl

When I have some time I’d like to look at making some more MQTT Bash scripts that could be used with file input.

The next step is to install the Mosquitto client on my Linux PC and Raspberry Pi:

sudo apt-get install mosquitto-clients

There are a number of MQTT brokers that can be used, both Home Assistant and Node Red have reliable brokers. The Mosquitto broker can also be loaded on a Linux, MacOS and Windows node.

Publish MQTT Data in Bash

The Mosquitto client can be used to both publish and subscribe to MQTT data.

For my router project, I wanted the: temperature (from a USB thermometer), idle time, % used space and available space. Below is the script that passed the data to the mosquitto_pub (publishing) tool:

#!/usr/ash
#
# mqtt_data.sh - send data to MQTT broker
#
# Get Data values
temp=$(/usbstick/temper.sh)
idle=$(vmstat | awk '{ if (NR==3) print $15}')
used=$(df | awk '{if (NR==4) printf "%.f\n", $5 }')
space=$(df | awk '{if (NR==4) printf "%.1f\n", $4/1000 }')

echo "$temp $idle $used $space"

# Publish Data
server="192.168.0.111"
pause=2

mosquitto_pub -h $server -t rtr_temp -m $temp
sleep $pause
mosquitto_pub -h $server -t rtr_idle -m $idle
sleep $pause
mosquitto_pub -h $server -t rtr_used -m $used
sleep $pause
mosquitto_pub -h $server -t rtr_space -m $space

I added a pause between each publish to let me watch the actions.

The cron utility can be used to schedule the running of the script. Once a minute is the fastest time available with cron, so if faster times are needed the script could cycle with a while loop.

Read/Subscribe to MQTT Data

The mosquitto_sub client tool can be used to read or subscribe to the data.

To look at the router points:

$ service="192.168.0.111"
$ mosquitto_sub -h $server -v -t rtr_idle -t rtr_temp -t rtr_used -t rtr_space

rtr_temp 26.25
rtr_idle 100
rtr_used 66
rtr_space 1.1

The -v (–verbose) option show output with the topic names and the values. Topics, -t option, can be put on the command line or passing in as a file.

For single point monitoring the Zenity utility can be used. This utility is typically preloaded on most Linux and Rasp Pi systems. A script to create a progress bar dialog with MQTT data is:

# Send MQTT values to a progress bar
( 
  while :; do
  msg=$(mosquitto_sub -h 192.168.0.111 -C 1 -t rtr_temp) 
  echo $msg
  echo "#$msg  Deg C"
  done
  ) | zenity --progress  --title="Router External Temperature"

Multipoint Progress Dialogs

The Zenity utility can only manage single point progress bars. For multiple point progress bars the YAD (Yet Another Dialog) package can be used.

To install YAD on Raspberry Pi’s and Ubuntu: sudo apt-get install yad

Below is some script that will show multiple points in a YAD dialog:

#!/usr/bash
#
# mqtt_bars.sh - Show multiple MQTT Topics on a Dialog
#
server="192.168.0.111"
topics=("rtr_idle" "rtr_temp" "rtr_used" )
scale=(100 40 100 )
units=( '%' 'degC' '%'  )
title="Router MQTT Points"

#Build topic and yad strings
yadstr=" "
for i in ${topics[@]}; do
  topstr="$topstr -t $i"
  yadstr="$yadstr --bar=$i"
done

echo "Press [CTRL+C] to stop..."
# Cycle thru 1 message at a time to YAD 
(
while : 
do 
  msg=$(mosquitto_sub -h $server -v -C 1 $topstr) 
  IFS=' ' read -a data <<< "$msg" 
  # match returned msg to order of bars, write value/label
  for i in "${!topics[@]} "
  do 
    if [ "${data[0]}" = "${topics[i]}" ]
    then 
      let j=i+1 ; # YAD indices start at 1
      # Rescale bar to defined scale
      barsize=$(bc <<< "${data[1]}*100/${scale[i]}")
      #barsize=100
      echo "$j:$barsize"
      echo  "$j:#${data[1]} ${units[i]}"
  fi
  done
done 
)  | yad --multi-progress $yadstr --title $title

Final Comments

There are probably lots of good 3rd party tools like Gnuplot that could be connected to mosquitto_sub to show real charts.

Bash Bar Charts for Text and Web Pages

This blog documents my notes on creating Bash text bar charts. The key topics are:

  • Create a simple horizontal bars (3 lines)
  • Create dynamic bars (~25 lines for an array of data values)
  • Add support for piped and command line data
  • Include options for: help, resizing, titles
  • Add colour and HTML support

A Horizontal Bar

To create a static bar an printf statements can be used. The seq {0..10} can be used to repeat an ASCII █ fill character 10 times.

$ printf 'LABEL: ' ; \
  printf '█%.0s' {1..10} ; \
  printf ' 10\n'

LABEL: ████████████████████ 10

Unfortunately the printf statement has some limitations on variable substitution. A simple workaround is to create a string and then eval it:

$ label="temp"; val=20;
$ bar="printf '█%.0s' {1..$val}" ; 
$ printf '\n%-5s ' $label; eval $bar ; printf ' %d\n' $val

temp  ████████████████████ 20

Coloured Bars

The tput setaf command can change the foreground, and tput setab is used for background colours. Colour codes are:

tput setab [1-7] # Set the background colour using ANSI escape
tput setaf [1-7] # Set the foreground colour using ANSI escape

Num  Colour    #define         R G B

0    black     COLOR_BLACK     0,0,0
1    red       COLOR_RED       1,0,0
2    green     COLOR_GREEN     0,1,0
3    yellow    COLOR_YELLOW    1,1,0
4    blue      COLOR_BLUE      0,0,1
5    magenta   COLOR_MAGENTA   1,0,1
6    cyan      COLOR_CYAN      0,1,1
7    white     COLOR_WHITE     1,1,1

To reset colours back to the defaults use: tput sgr0

An example to print a red bar and a stack of bars:

$ printf '\nLABEL: ' ; \
   tput setaf 1 ;\
   printf '█%.0s' {1..10} ; \
   printf ' 10\n'

LABEL: ██████████ 10
$ printf '\n 3 Stacked Bars: ' ; \
   tput setaf 1 ;\
   printf '█%.0s' {1..10} ; \
   tput setaf 2 ;\
   printf '█%.0s' {1..8} ; \
   tput setaf 4 ;\
   printf '█%.0s' {1..3} ; \
   printf ' 10+8+3=21\n'

 3 Stacked Bars: █████████████████████ 10+8+3=21

Dynamic Bars

The next step is to create a script that dynamically updates the bars. The tput clear command will clear the terminal screen keep the data and bars in the same location. The script below will dynamically show the CPU temperature, idle time and 2 random values with a 10 second update time.

#!/bin/bash
# 
# cpu_bars.sh - Show new data every 10 seconds
#
while :; do
    # Get data values
    CPUtemp=$(sensors | grep CPU | awk '{print substr($2,2,4)}')
    CPUidle=$(iostat | awk '{if (NR==4) print $6}')
    Random1=$((1+ $RANDOM % 100))
    Random2=$((1+ $RANDOM % 100))

    labels=( CPUtemp CPUidle Random1 Random2)
    values=( $CPUtemp $CPUidle $Random1 $Random2)
    units=( degC % psi mm)

    # Show a title
    tput clear
    printf " %10s " "" 
    tput setaf 7; tput smul;
    printf "%s\n\n" "Show CPU Data ($(date +%T'))"
    tput rmul;

    # cycle thru data and show a label, 
    for index in "${!labels[@]}"
    do
          tput setaf $(expr $index + 1); # don't use 0 (black) 
          printf " %10s " "${labels[index]}"
          eval "printf '█%.0s' {1..${values[index]%.*}}"
          printf " %s %s\n\n" ${values[index]} ${units[index]}
    done
    sleep 10
done

This script is run by: bash cpu_bars.sh .Typical output is below.

Bars in HTML

The ANSI colours are not supported in HTML, so instead HTML tags and style properties can be used.

It is important to use <pre> tags for Bash text output. Code to create two static bars in HTML would be:

$ (printf "<h1>A Bar from Bash</h1>\n" 
 printf "<pre><span style='font-size:24px;color:red'}>\n"
 printf 'LABEL1: ' ; printf '█%.0s' {1..10} ; printf ' 10\n'
 printf "</pre></span>\n") > bar1.htm

$ cat bar1.htm
<h1>A Bar from Bash</h1>
<pre><span style='font-size:24px;color:red'}>
LABEL1: ██████████ 10
</pre></span>

The script cpu_webbars.sh creates HTML output for an array of calculated values:

#!/bin/bash
# 
# cpu_webbars.sh - Show bars in HTML
#

# Get data values
CPUtemp=$(sensors | grep CPU | awk '{print substr($2,2,4)}')
CPUidle=$(iostat | awk '{if (NR==4) print $6}')
Random1=$((1+ $RANDOM % 100))
Random2=$((1+ $RANDOM % 100))

labels=( CPUtemp CPUidle Random1 Random2)
values=( $CPUtemp $CPUidle $Random1 $Random2)
units=( degC % psi mm)
colors=(red blue green magenta)

# Show a title
printf "<h1><center>Show CPU Data ($(date '+%T'))</center></h1>\n"
# cycle thru data and show a label, 
for index in "${!labels[@]}"
  do
  printf "<pre><span style='font-size:18px;color: ${colors[index]} '}>\n"
  printf " %10s " "${labels[index]}"
  eval "printf '█%.0s' {1..${values[index]%.*}}"
  printf " %s %s\n\n" ${values[index]} ${units[index]}
  printf "</pre></span>\n"
done

This script can be run and outputted to an file: bash cpu_webbars.sh > test.htm

Once the basic HTML output is working its possible to add headers and footers to make a more complete page:

header.htm > test.htm ; \
cat cpu_webbars.sh >> test.htm ; \
cat footer >> test.htm 

Piping Data to Bars

A script (hbar0.sh) will read the piped data and then create an array (data) of labels and values. The data array is cycled through and labels and bars are shown with a randomized colour:

#!/bin/bash
# hbar0.sh - Read in piped data  and plot bars
#     format: label,value;label2,value2;  and plot bars
#
input=$(< /dev/stdin) ; # read piped data
# remove new lines in files, and check for ";" after data pair 
input=$(echo $input | tr -d '\n')
IFS=';' read -r -a data <<< $input
printf "\n" 
for element in "${data[@]}"pete@lubuntu:~/Writing/Blog/text_bars
do 
  # make at array of each data element
  IFS=',' read -r -a datapt <<< $element
  # add a random color
  tput setaf $((1+ $RANDOM % 7))
  # print the label, bar and value
  printf " %10s " "${datapt##*[0]}"
  bar="printf '█%.0s' {1..${datapt[1]}}"
  eval $bar
  printf " %s\n\n" ${datapt[1]} 
  tput rmso ; # exit color mode   
done

The script can be tested with piped data:

$ echo "temp,33;pressure,44" | bash hbar0.sh

 temp       █████████████████████████████████ 33 

 pressure   ████████████████████████████████████████████ 44 

A data file can also be passed in using the cat command:

$ cat data0.txt 
temp,44;
humidity,33;
pressure,15;
wind spd,33;
wave ht,3;
$ cat data0.txt | bash hbar0.sh

       temp ████████████████████████████████████████████ 44

   humidity █████████████████████████████████ 33

   pressure ███████████████ 15

   wind spd █████████████████████████████████ 33

    wave ht ███ 3

Removing ANSI Colors

Terminal applications use ANSI color codes which unfortunately is not support on Web pages.

ANSI color codes can be removed from files and strings by:

# Strip out ANSI color codes:
cat infile | sed 's/\x1b\[[0-9;]*m//g' > outfile
$ echo "temp,33;pressure,44" | bash hbar0.sh > hbar0.txt
$ cat hbar0.txt

       temp █████████████████████████████████ 33

   pressure ████████████████████████████████████████████ 44

$ cat hbar0.txt | sed 's/\x1b\[[0-9;]*m//g'

       temp █████████████████████████████████ 33

   pressure ████████████████████████████████████████████ 44

A Final App

With the basics in place I was able to create an app that would support scaling, titles, units, custom colour and web output:

$ ./hbars
usage: hbars [data] [option] 
  -h --help     print this usage and exit
  -c --color    set color to all bars (default 7=white)
                 (0-grey,1-red,2=green,3=yellow,4=blue,5=magenta,6=cyan,7=white)
  -p --pretty   add different colors to bars (-c overrides)
  -t --title    top title
  -w --width    max width of bar (default 50)
  -W --Web      make output HTML formatted
  -f --fontsize fontsize for web output (default 24)

 examples:
   echo 'temp,33,C;pressure,14,psi' | ./hbars -t Weather -p -w 40 
   ./hbars -t Weather -p -w 40  'temp,33;pressure,14' 
   cat data.txt | ./hbars -W -f 24 -t 'Raspi Data' > data.htm

The code:

#!/bin/bash
#
# hbars.sh - show some text bars
#   pass data as:  label1,value1,unit1,color1; label2,value2,unit2,colour2; ....  
#
width=50
title=""
color=7
pretty=False
web=False
font=24

usage() { 
  echo "usage: hbars [data] [option] "
  echo "  -h --help     print this usage and exit"
  echo "  -c --color    set color to all bars (default 7=white)"
  echo "                 (0-grey,1-red,2=green,3=yellow,4=blue,5=magenta,6=cyan,7=white)"
  echo "  -p --pretty   add different colors to bars (-c overrides)"
  echo "  -t --title    top title"
  echo "  -w --width    max width of bar (default 50)"
  echo "  -W --Web      make output HTML formatted"
  echo "  -f --fontsize fontsize for web output (default 24)"
  echo ""
  echo " examples:"
  echo "   echo 'temp,33,C;pressure,14,psi' | ./hbars -t Weather -p -w 40 "
  echo "   ./hbars -t Weather -p -w 40  'temp,33;pressure,14' "
  echo "   cat data.txt | ./hbars -W -f 24 -t 'Raspi Data' > data.htm"
  echo ""

  exit 0
}
# Show help usage if no pipe data and no cmd line data
if [ -t 0 ]  && [ $# -eq 0 ] ; then
  usage
fi
# Check for command line options
while getopts "hpc:t:w:Wf:" arg; do
  case "$arg" in
    h) usage ;;
    c)  color=$OPTARG ;;
    p)  pretty=True; icolor=0 ;;
    t)  title=$OPTARG ;;
    w)  width=$OPTARG ;;
    W)  web=True;;
    f)  font=$OPTARG ;;
  esac
done
#------------------------------------------------
# Setup formatting for text, Web and color
# -----------------------------------------------
if [[ ${color} != 7 && ${pretty} = True ]]; then
  pretty=False
fi
colidx=0

setcolors() {
if [ $web = True ]; then
  colors=(gray red green yellow blue magenta cyan white)
  titlebold="echo '<h1>'"
  titlereset="echo '</h1>'"
  #color_set='echo "<span style=font-size:$(font)px  >" ' 
  #color_set="printf '<span  style=\"font-size:$(font)px;color:${colors[colidx]}\" >'" 
  color_set="printf '<pre><span style=\"color:${colors[colidx]} ; font-size:${font}px \" >'" 
  color_rs="echo '</span></pre>'"
else
  colors=(0 1 2 3 4 5 6 7 )
  titlebold="tput bold; tput smul"
  titlereset="tput rmul; tput rmso"
  color_set="tput setaf ${colors[colidx]}"
  color_rs="tput rmso"
fi
}
setcolors
#----------------------------------------------
# Get data, check if stdlin, file, if not assume string
#----------------------------------------------
if [ -p /dev/stdin ]; then
        lastarg=$(< /dev/stdin)
else
	lastarg=$(echo "${@: -1}")
	if test -f "$lastarg"; then
	  lastarg=$(<$lastarg)
	fi
fi
# Cleanup the input data
lastarg=$(echo $lastarg | sed 's/\n/;/g; s/  / /g; s/\t//g; s/;;/;/g')
IFS=';' read -r -a array <<< $lastarg

# ensure that there is some data
if [[ ${array} == 0 ]]; then
  echo "No data found"
  exit 0
fi
echo "input:$lastarg"
#exit 0
#------------------------------------
# Get max value and max label length
#------------------------------------
maxval=0
maxlbl=10
#echo "array:${array[@]}"
for element in "${array[@]}"
do 
  IFS=',' read -r -a datapt <<< $element
  if (( $(echo "$maxval < ${datapt[1]}" |bc -l) )); then
	maxval=${datapt[1]}
  fi
  if (( $(echo "$maxlbl < ${#datapt[0]}" |bc -l) )); then
	maxlbl=${#datapt[0]}
  fi
done
#---------------------------------
# Print Title - use bold/underline
#---------------------------------
if [[ ! -z $title ]]; then
  printf "\n %${maxlbl}s " " "
  eval $titlebold
  printf "%s" "${title}" ; printf "\n\n"
  eval $titlereset
fi
#------------------------------------
# Cycle thru data and build bar chart
#------------------------------------
for element in "${array[@]}"
do
# check data values
  IFS=',' read -r -a datapt <<< $element
  # check for empty records
  if [ ${#datapt[0]} = 0 ]; then 
    break
  fi
  label=${datapt[0]}
  if [[ ${label} != "-t*" ]]; then 
	  val=${datapt[1]}
	  sval=$(bc <<< "$width * $val / $maxval")

	  # add color, use 4th item if available
	  if [[ ${#datapt[@]} > 3 && $pretty = False ]]; then
		icolor=${datapt[3]}
	  fi
	  if [[ $pretty = True ]] ; then
		let colidx++
		if [ $colidx -gt 7 ]; then
		  let colidx=1
		fi
	  elif [[ ${#datapt[@]} > 3 ]]; then
		colidx=${datapt[3]}
	  else
	  	colidx=$color
	  fi
	  setcolors
          eval $color_set
          printf " %${maxlbl}s " "$label"
	  bar="printf '█%.0s' {1..$sval}"
	  eval $bar; 
	  # add value and units if available
	  units=""
	  if [[ ${#datapt[@]} > 2 ]]; then
		units=${datapt[2]}
	  fi
	  printf " %d %s\n\n" $val "$units"
	  eval $color_rs 
  fi
done

SQL Output to Bars

With the base code I was able to start doing some more complicated actions, like piping SQL SELECT output. Below is an example from Sqlite3. The first step is to format the SQL output to: label,value;

pete@lubuntu:$ sqlite3 $HOME/dbs/someuser.db "select fname,age from users limit 4"
Brooke|18
Leah|18
Pete|100
Fred|77
pete@lubuntu:$ sqlite3 $HOME/dbs/someuser.db "select fname,age from users limit 4" \
>  | tr  '|' ',' | tr '\n' ';'
Brooke,18;Leah,18;Pete,100;Fred,77;
pete@lubuntu:$ sqlite3 $HOME/dbs/someuser.db "select fname,age from users limit 4" \ > | tr '|' ',' | tr '\n' ';' | ./hbars -t "Sqlite3 Users" -p Sqlite3 Users Brooke █████████ 18 Leah █████████ 18 Pete ██████████████████████████████████████████████████ 100 Fred ██████████████████████████████████████ 77

Web Page with Bars

Support was added for fixed chart width, engineering units and custom line colours. HTML <center> tags were used on the title.

$ cat data.txt
temp,44,degC,1;
humidity,33,%,4;
air pressure,88,mm Hg,5;
rain/precipation,6,mm,6;

$ cat data.txt | ./hbars -W -f 24 -w 50 -t '<center>Raspi Data</center>' > data.htm

Final Comments

Horizontal bars are relatives easy to create in Bash. Unfortunately showing vertical bars and Line charts will require a different approach

A Web Server in 1 Line of Bash Code

For people who do a lot of work with command line tools or Bash code, having a Bash web server could be very handy.

I was really amazed that in one line of Bash code I was able to create web servers that could:

  • send the output from a bash command directly to a browser page
  • create diagnostic pages using standard Linux tools
  • create pages that view Rasp PI GPIO pins
  • create a page to toggle a PI GPIO pin

One Line Web Servers

There are number of 1 line web servers, these are minimal offerings that exist in most of the programming languages.

A Bash web server can be created using the nc or netcat, the networking utility:

while true; do { \
  echo -ne "HTTP/1.0 200 OK\r\nContent-Length: $(wc -c <index.htm)\r\n\r\n"; \
  cat index.htm; } | nc -l -p 8080 ; \ 
done

This Bash statement echo’s to port 8080, the output is an HTTP header with the file content length defined. The cat command is used to show the HTML file.

This 1 line Bash example shows a single page (index.htm) which isn’t overly useful, there are other web server options that would work much better.

Where a Bash web server really stands out is in its ability to execute command line utilities or scripts and send the results to a web client.

Bash Web Server Calling Bash Commands

The output from command line utilities like iostat can be sent directly to a web client:

while true;
  do echo -e "HTTP/1.1 200 OK\n\n$(iostat)" \
  | nc -l -k -p 8080 -q 1; 
done

There are 2 important options that need to be set on nc, and they are: -k (this keeps the connection open after the first connection) and -q 1 (this closes the connection after 1 seconds, so another connection can occur). Depending on the complexity of the script that is used the -q timing may need to be adjusted.

The web page for the iostat command will look like:

Multiple Commands with Headings

Comments and multiply command line utilities can be defined as a variable that can be passed to the Bash Web server.

The figlet utility can be used to create custom ASCII headings, this is useful if you want to keep things simple without using HTML syntax. To install figlet in Ubuntu enter: sudo apt-get install figlet .

An example of using figlet headings and the sensors and vmstat utility is:

title1=$(figlet Sensors)
cmd1=$(sensors | sed -e 's/\°/ /g') # browser has problem with degrees, so remove
title2=$(figlet VMStat)
cmd2=$(vmstat)
thebody="$title1\n$cmd1\n$title2\n$cmd2"

while true;
  do echo -e "HTTP/1.1 200 OK\n\n$thebody" \
  | nc -l -p 8080 -q 1; 
done

Bash Web Server with Raspberry Pi GPIO

For many Raspberry Pi projects monitoring the status of the GPIO (General Purpose Input/Output) pins is quite important.

The Raspberry Pi gpio utility can be used to show the present status with the readall option:

pi@raspberrypi:~/pete $ gpio readall
 +-----+-----+---------+------+---+---Pi 3B--+---+------+---------+-----+-----+
 | BCM | wPi |   Name  | Mode | V | Physical | V | Mode | Name    | wPi | BCM |
 +-----+-----+---------+------+---+----++----+---+------+---------+-----+-----+
 |     |     |    3.3v |      |   |  1 || 2  |   |      | 5v      |     |     |
 |   2 |   8 |   SDA.1 | ALT0 | 1 |  3 || 4  |   |      | 5v      |     |     |
 |   3 |   9 |   SCL.1 | ALT0 | 1 |  5 || 6  |   |      | 0v      |     |     |
 |   4 |   7 | GPIO. 7 |  OUT | 0 |  7 || 8  | 0 | IN   | TxD     | 15  | 14  |
 |     |     |      0v |      |   |  9 || 10 | 1 | IN   | RxD     | 16  | 15  |
 |  17 |   0 | GPIO. 0 |   IN | 0 | 11 || 12 | 0 | IN   | GPIO. 1 | 1   | 18  |
 |  27 |   2 | GPIO. 2 |  OUT | 0 | 13 || 14 |   |      | 0v      |     |     |
 |  22 |   3 | GPIO. 3 |   IN | 0 | 15 || 16 | 0 | IN   | GPIO. 4 | 4   | 23  |
 |     |     |    3.3v |      |   | 17 || 18 | 0 | IN   | GPIO. 5 | 5   | 24  |
 |  10 |  12 |    MOSI | ALT0 | 0 | 19 || 20 |   |      | 0v      |     |     |
 |   9 |  13 |    MISO | ALT0 | 0 | 21 || 22 | 0 | IN   | GPIO. 6 | 6   | 25  |
 |  11 |  14 |    SCLK | ALT0 | 0 | 23 || 24 | 1 | OUT  | CE0     | 10  | 8   |
 |     |     |      0v |      |   | 25 || 26 | 1 | OUT  | CE1     | 11  | 7   |
 |   0 |  30 |   SDA.0 |   IN | 1 | 27 || 28 | 1 | IN   | SCL.0   | 31  | 1   |
 |   5 |  21 | GPIO.21 |  OUT | 0 | 29 || 30 |   |      | 0v      |     |     |
 |   6 |  22 | GPIO.22 |  OUT | 0 | 31 || 32 | 0 | OUT  | GPIO.26 | 26  | 12  |
 |  13 |  23 | GPIO.23 |  OUT | 0 | 33 || 34 |   |      | 0v      |     |     |
 |  19 |  24 | GPIO.24 |  OUT | 0 | 35 || 36 | 0 | IN   | GPIO.27 | 27  | 16  |
 |  26 |  25 | GPIO.25 |  OUT | 0 | 37 || 38 | 0 | OUT  | GPIO.28 | 28  | 20  |
 |     |     |      0v |      |   | 39 || 40 | 0 | OUT  | GPIO.29 | 29  | 21  |
 +-----+-----+---------+------+---+----++----+---+------+---------+-----+-----+
 | BCM | wPi |   Name  | Mode | V | Physical | V | Mode | Name    | wPi | BCM |
 +-----+-----+---------+------+---+---Pi 3B--+---+------+---------+-----+-----+

Rather than passing the Bash commands as a string, a small Bash script (web_body.sh) file can be created. This file will show the time and then call the gpio readall command.

#!/bin/bash
# web_body.sh - Show the time and PI GPIO pins

date $T

echo "$(gpio readall)"

The online Bash web server command is:

while true; do { \
  echo -ne "HTTP/1.1 200 OK\r\n"; sh web_body.sh; } \
  | nc -l -k -q 2 8080; \
done

The Web page for this script will look something like:

Send GPIO Writes from the Address Bar

Client side GET requests can be simulated on the browser address bar.

For example entering gpio write 7 1 on the address bar sents the Bash Server a GET request.

The HTTP request encodes spaces, so a space appears as a %20.

Bash code can be added to look for specific messages. In this case the “gpio write 7 1” or “gpio write 7 0” message can be search for, and if it is found then that exact message can be executed.

The Bash code can now be modified to ; look for the “GET gpio” message, then decode any HTTP %20 characters to spaces, next parse out the string to get the gpio message, and finally execute the required command. The code is below:

while true; 
  do { echo -ne "HTTP/1.1 200 OK\r\n"; sh web_body.sh; } | \
  nc -l -k -q 5 8080 | grep "GET /gpio" | \
  sed -e 's/%20/ /g' | \
  eval $( awk '{print substr($0,6,15) }') ;
done

With the new code, the gpio write is executed and the result can be seen in the web page.

Create an HTML Form

Entering commands on the command line works but it’s crude, a better way is to create an HTML Form.

The Bash web server code can remain exact the same as in the earlier example.

The original script (web_body.sh) file can be modified to made the output HTML format and three forms can be included. The first and second form will define the GET actions to turn on or off the GPIO pin and the third form will be used to refresh the page to check for GPIO changes.

#!/bin/bash
# web_body.sh - Show the time and PI GPIO pins
#             - Use HTML instead of text output
#             - Add forms for GPIO on/off, and a refresh
echo "
<!DOCTYPE html><html><head>
</head><body>
<h1>Bash Commands in a Web Page</h1>
<h2>Toggle Pin 7 On/Off</h2>
<form action='gpio write 7 0'>
 <input type='submit' value='OFF'> 
</form> 
<form action='gpio write 7 1'>
 <input type='submit' value='ON'>
</form>
<form action=''>
 <input type='submit' value='Refresh Page'>
</form>
<pre>
"

date $T

echo "$(gpio readall)"

echo "</pre></body></html>"

The client web page is now:

After turning on or off the GPIO pin, a refresh of the web page is required to see the new status.

Final Commands

A Bash Web Server is a quick and easy solution for viewing the output from Bash scripts and commands. I especially like the fact that I don’t need to install any special software and I don’t need to write any HTML code.

It is important to note that the number of concurrent connections is very low, (1/sec if the nc -q option is 1).

A Bash Web Server supports client side GET and POST requests, however for complex requirements the Bash code could start to get messy quickly and it would probably be best to look at another solution.

Simple TCP/UDP Bash Apps

There are some great communication protocols (MQTT, RabbitMQ, ReDis …) that are excellent for passing data between nodes.

For applications where you only need to do simple communications a couple of lines of Bash can be used with TCP or UDP sockets.

In this blog I wanted to document UDP/TCP communications using Linux Bash commands to:

  • define periodic 1-way communications
  • use progress bars to show data from remote nodes
  • remotely send commands to a Raspberry Pi
  • setup simple TCP backdoors

NC (NetCat) – for TCP and UDP Connections

In theory you should be able to create a input read via something like:

echo $(read < /dev/udp/127.0.0.1/9999)

Then do a write using:

echo "some text" > /dev/udp/127.0.0.1/9999)

Unfortunately Linux device connections are not fully reliable, especially on the read or listening side. However the write component appears to be fairly solid. Luckily there is a solid solution using the nc (NetCat) command line utility. The nc utility is typically preloaded on most Linux systems.

The nc utility supports both UDP (-u option) and TCP (default) connections.

To setup a UDP listener, use the IP address of the listener node, and select the -k option to allow multiple connections to occur:

nc -u -l -k 192.168.0.111  9999

For this example the listener’s IP is: 192.168.0.111, and port 9999 is used.

To do manual writes from the command line, enter:

 nc -u  192.168.0.111  9999

To send data from a script there are two methods, either using nc or writing to the device:

# writing via nc, -w0 send only 1 message 
echo "456" | nc -u -w0 192.168.0.111  9999
# writing via device:
echo "456" > /dev/udp/192.168.0.111/9999

Multiple Writes and Zenity Progress Dialogs

Zenity is command line dialog utility that is typically preinstalled on most versions of Linux.

The data that is sent to the UDP listener can be piped to Zenity progress bar:

nc -u -l -k 192.168.0.111  9999 | zenity --progress --title="Remote Data"

A script to send seconds every second would be:

#!/bin/bash
echo "Press [CTRL+C] to stop..." 
( 
while : 
do
# $(date +'%S') seconds" | nc -u -w0 192.168.0.111 9999
 echo "$(date +'%S')" | nc -u -w0 192.168.0.111 9999
 sleep 1 
done 
)

When the script echos an integer the progress bar will be updated with the integer value. An echo string starting with a “#” will update the text above the bar.

The progress bar is from 0-100%, but the integer value can be re-scaled to make the information clearer. For example to re-scale 0-60 secs to 0-100:

echo "$(date +'%S')*100/60" | bc | nc -u -w0 192.168.0.111 9999

YAD – for Multiple Progress Bars

YAD (Yet Another Dialog) is a command line GUI utility that offers a little more functionality than Zenity. To install YAD on Raspberry Pi’s and Ubuntu: sudo apt-get install yad

A bash command with a UDP listener with YAD 2-bars would be:

nc -u -l -k 192.168.0.111  9999 | yad --multi-progress \
  --bar="CPU Idle" --bar="CPU Temp" --title="Remote CPU Info"

The CPU Idle Time can be found by:

top -n 1 | grep %Cpu | awk '{print $8}'
93.8

The CPU Temperature on a PC can be found by:

sensors | grep CPU | awk '{print substr($2,2,4)}'
44.0

A script to send the CPU Idle Time and Temperature to the UDP listener is:

#!/bin/bash
echo "Press [CTRL+C] to stop..." 
( 
while : 
do
 cpuidle=$(top -n 1 | grep %Cpu | awk '{print $8}') 
 echo "1:"$cpuidle | nc -u -w0 192.168.0.111 9999

 cputemp=$(sensors | grep CPU | awk '{print substr($2,2,4)}')
 echo "2:"$cputemp | nc -u -w0 192.168.0.111 9999
 echo "2:#"$cputemp" Deg C" | nc -u -w0 192.168.0.111 9999
 sleep 5 
done 
)

For YAD multiple progress bars, an echo of 1: is for bar 1, 2: is for bar 2 etc. Echo-ing “2:# ” updates the text for the 2nd bar.

NC is not the same on Rasp Pi

I found that on the Rasp Pi the nc listening functions would not pass any information to bash scripts. Manual mode still works to view messages, but the messages can’t be piped to other commands.

This mean that things like the Zenity and YAD progress bars would not work on a Raspberry Pi. For many applications this may not be a big problem because the Rasp Pi can still send information via nc.

NC vs. NCAT

The ncat utility is very similar to nc but it offers the ability to run commands. By default nc is preloaded on most systems, but ncat needs to be installed. Installing ncat will vary based on your OS.

The ncat utility allows you to make backdoors so be careful of its use.

To create a backdoor simply (via TCP), define the ncat -c (execute command option) to be /bin/bash:

ncat -l -k 192.168.0.108  9999 -c /bin/bash

If on a remote node you enter: ncat 192.168.0.108 , you can start typing commands that are run on the remote node with the results echoing back. Very cool for test system but super dangerous for real systems.

Remotely Toggle a Rasp Pi GPIO Pin

Rather than opening up the system totally fixed commands can be defined. For example to toggle pin 7 on a Rasp Pi. A listener script is run:

ncat -l -k 192.168.0.108  9999 -c "gpio toggle 7"

A remote button GUI script could be used in conjunction with the listener script to toggle the GPIO pin:

#!/bin/bash
#
# Toggle a Rasp Pi GPIO pin

rc=1 # OK button return code =0 , all others =1
while [ $rc -eq 1 ]; do
  ans=$(zenity --info --title 'Remote Connect to Pi' \
      --text 'Toggle GPIO Pin' \
      --ok-label Quit \
      --extra-button TOGGLE \
       )
  rc=$?
  echo "${rc}-${ans}"
  echo $ans
  if [[ $ans = "TOGGLE" ]]
  then
        echo "Toggle Pin"
        nc -w0 192.168.0.108  9999
  fi
done

Send a Command String to Run Remotely

For this example the Rasp Pi is setup to be TCP listener, and the command (-c option) is /bin/bash, so this allows the remote PC to send custom commands:

ncat -l -k 192.168.0.108  9999 -c /bin/bash

On the remote system a bash script is created with 2 buttons and the custom commands are sent to the Pi to run:

#!/bin/bash
#
# Toggle two Rasp Pi GPIO pins

rc=1 # OK button return code =0 , all others =1
while [ $rc -eq 1 ]; do
  ans=$(zenity --info --title 'Remote Connect to Pi' \
      --text 'Toggle GPIO Pins' \
      --ok-label Quit \
      --extra-button Pin2 \
      --extra-button Pin7 \
       )
  rc=$?
  echo "${rc}-${ans}"
  echo $ans
  if [[ $ans = "Pin2" ]]
  then
        echo "gpio toggle 2" | nc -w0 192.168.0.108  9999
  elif [[ $ans = "Pin7" ]]
  then
        echo "gpio toggle 7" | nc -w0 192.168.0.108  9999
  fi
done

On systems other than Rasp Pi, the nc command can also be used to run remote programs by:

nc -u -l -k 192.168.0.111  9999 | awk '{ system($1 " " $2 " " $3 " " $4)}'

Final Comments

In this blog I’ve kept things focused on TCP/UDP communications with bash script but you could easily include Arduino, Python and Node-Red as either clients or servers.

Zenity: Command line Dialogs

Zenity is command line GUI creator that has been around since 2012, and it is pre-installed on most versions of Linux, including Raspberry PI’s. Zenity is also available for MacOS and Windows.

I came across it on a recent project and I wanted to document some of my code. My examples include:

  • CPU stats on a dialog – 1 line of Bash script
  • Show a web page in a dialog – 1 line
  • Create a 4 button PI Rover control – ~ 25 lines
  • Dynamic bar of CPU core temperature – 7 lines
  • Show CSV or SQL data in a list dialog – 1 line
  • Form to insert user data in an SQL database – ~7 lines

What is Zenity?

Zenity is a command line dialog creator. I found it pretty quick to pick up and it works well for simple Bash scripts. Zenity supports:

  • Basic forms
  • Calendar dialogs
  • Color selection dialogs
  • File Selection Dialogs
  • List Dialog
  • Message and Notification Dialog
  • Progress bars and Scales
  • Text Entry and Text Information Dialogs

Message Dialogs

Message dialogs can be created for errors, info, questions or warnings. The difference is the icon that shows up (and Ok/Cancel for the question dialog).

The Bash code to get the instantaneous CPU idle time would be:

pi@raspberrypi:~ $ # Run top once and look for the line with %Cpu
pi@raspberrypi:~ $ top -n 1 | grep Cpu
%Cpu(s):  7.4 us,  3.6 sy,  0.0 ni, 88.8 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st

pi@raspberrypi:~ $ # Get the 8th item from the grep

pi@raspberrypi:~ $ top -n 1 | grep %Cpu | awk '{print $8}'
88.8

The bash code to show the CPU idle time in a info dialog is:

zenity --info --text=$(top -n 1 | grep %Cpu | awk '{print $8}') --title="CPU Idle Time"

Message Dialogs with Custom Font

Text font and size can be modified in message dialogs, using the Pango Markup Language syntax. Pongo is similar to HTML. The <span></span> set of tags is used to encode font and color definitions, For example:

zenity --warning --text='<span font="32" foreground="red">HIGH Temperature</span>' --title="HDD Check"

Unfortunately only Message Dialog texts can changed, so text in dialogs like list, scale, and progress can’t not have their fonts changed.

Web Pages in a Text-Info Dialog

Text or HTML files can be passed to a text-info dialog:

zenity --text-info --title="Background Reading" --html --url="https://developer.gnome.org"

A checkbox can be added, and the user feedback can be read by the bash script:

#!/bin/sh
# show web page in a dialog with a next step action
#
theurl="https://developer.gnome.org"

zenity --text-info --title="Background Reading" --html --url=$theurl \
       --checkbox="I read it...and I'm good to go"
rc=$?
echo $rc
case $rc in
    0)
        echo "Start some next step"
	# next step
	;;
    1)
        echo "Stop installation!"
	;;
   -1)
        echo "An unexpected error has occurred."
	;;
esac

It is important to note that the text-info dialog should be used for simple web pages. There is no Javascript support and web links will launch the default web browser with requested page.

Refreshing Message Dialogs

Of all the Zenity dialogs, only the Progress dialog supports a method to update text on an open dialog. A workaround is to use the –timeout option to close the dialog and then redisplay it with the new data.

rc=5
while [[ $rc -eq 5 ]];
do 
  zenity --info --text=$(date +'%S' ) \
  --title="Seconds Timer Test" --timeout=5 --ok-label Quit $ zenity --info \
  --text=$(date +'%S' )   --title="Seconds Timer Test" 2>/dev/null

  rc=$?	
  echo $rc
done 

This “timeout and redraw” method is ugly because the window always positions in the middle of the screen and this can be quite annoying. Unfortunately Zenity does not support any top/left positioning options.

The xdotool could be used find the zenity window id and then position it, but this would need to be done in another script. (It can’t be done in the same script because the zenity line doesn’t complete until either it times out or OK is pressed). The xdotool script would be:

pid=$(xdotool search -onlyvisible -name myzenitywindownname)
xdotool windowmove $pid 0 0

If you need dynamically updated text on a dialog I think that it would be best to use another tool, (Python, YAD etc.).

Info Dialog with Extra Buttons – Pi Rover Controls

It’s possible to add some extra button to an info dialog. Below is an example where a Raspberry Pi Rover is controlled with a zenity multi-button info dialog. Note: the pins will vary with your setupArduinoIt’s possible to add some extra button to an info dialog. Below is an example where a Raspberry Pi Rover is controlled with a zenity multi-button info dialog. Note: the pins will vary with your setup


#!/bin/bash
#
# rover.sh - Rover Controls with Multiple Button Dialog
# Define GPIO pins for the motors motorL=7 motorR=11 rc=1 # OK button return code =0 , all others =1 while [ $rc -eq 1 ]; do ans=$(zenity --info --title 'Drive a Rover' \ --text 'Motor Action' \ --ok-label Quit \ --extra-button FORWARD \ --extra-button STOP \ --extra-button LEFT \ --extra-button RIGHT \ ) rc=$? echo "${rc}-${ans}" echo $ans if [[ $ans = "FORWARD" ]] then echo "Running the Rover" gpio -1 write $motorL 1 ; gpio -1 write $motorR 1; elif [[ $ans = "STOP" ]] then echo "Stopping the Rover" gpio -1 write $motorL 0 ; gpio -1 write $motorR 0; elif [[ $ans = "LEFT" ]] then echo "Rover turning Left" gpio -1 write $motorL 1 ; gpio -1 write $motorR 0; elif [[ $ans = "RIGHT" ]] then echo "Rover turning RIGHT" gpio -1 write $motorL 0 ; gpio -1 write $motorR 1; fi done

The script can be run by: bash rover.sh

Below is the dialog and the rover.

Progress Bars – Show Dynamic Values

A Zenity progress dialog can show dynamic updates with scripts that define steps using sleep statements. When the step outputs a value the process bar is updated. The text on the progress dialog is changed by outputting a text string starting with a # character.

A 3-step example would be:

(
echo "33"; echo "# 1/3 done" ; sleep 5; \
echo "66"; echo "# 2/3 done" ; sleep 5; \
echo "100";echo "# Finished"  \
) | zenity --progress --title="3 step test"

The progress dialog can use a bash for or while statement. The progress dialog can be passed both new text and the value. A text string starting with # is interpreted as the new text. A number string is interpreted as the progress bar value.

Below is an example where a value is counted from 1 to 100:

( for i in `seq 1 100`; do echo $i; echo "# $i";  sleep 1; done ) | zenity --progress

The next thing I tested is a dialog that runs indefinitely (or until you hit “Control-C”). It’s import to note that the progress bar is from 0-100, so scaling your value may be required. An example of scaling a time from 0-60 would be:

echo "$(date +'%S')*100/60" | bc

An example to show seconds in a dialog would be:

#!/bin/bash
# show_sec.sh - progress dialog to show seconds
echo "Press [CTRL+C] to stop..." 
( 
  while :; do 
  echo "# $(date +'%S')" 
  # Scale 0-60 to 0-100 
  echo "$(date +'%S')*100/60" | bc
  sleep 1 
  done 
  ) | zenity --progress  --title="Show Time in Seconds"

To run this script: bash show_sec.sh

A more useful dialog would be to show the CPU temperature:

#!/bin/sh 
# show_cpu_temp.sh - Progress Dialog to show CPU temperature
# 
echo
 "Press [CTRL+C] to stop..."
(
while :; do 
  echo "# $(sensors | grep CPU)" 
  sensors | grep CPU | awk '{print substr($2,2,4) }' 
  sleep 5 
done ) | zenity --progress --title="CPU Temperature"  

List Dialog – Show CSV/SQL Data

If you are working with a simple known data set then the List Dialog might be a good fit.

The List Dialog expects the data to be a sequential list, so a 2 column example of static data would be:

zenity --list \
  --title="2 Column Example" \ 
  --column="Month" --column="Sales" \
   Jan 100 Feb 95 Mar 77 Apr 110 May 111

Text and CSV files can also be used in Zenity lists. The first step is to convert the file into a single column of data. This can be done with the tr statement. For the example below the comma (,) is replaced with a newline (\n) character:

$ cat lang.txt
Brazil,Brasilia,Portuguese
England,London,English
France,Paris,French
Germany,Berlin,German

$ cat lang.txt | tr ',' '\n'
Brazil
Brasilia
Portuguese
England
London
English
France
Paris
French
Germany
Berlin
German

Now the sequential data can be passed into a Zenity list:

cat lang.txt | tr ',' '\n' | zenity --list \
  --title="Country Info" \
  --column="Country" --column="Capital" --column="Language"

Once you have some Zenity and Bash basics down you can some fairly advanced operations. Below is a 1-line example that uses awk to parse out specific fields (1 and 3) and then the user selected output is echo-ed.

awk -F "\"*,\"*" '{print $1 "\n" $3}' pidata.csv  | \
  echo $(zenity --list --column="field1" --column="field3" --print-column=ALL)

Similarly to data from an SQL query can be show. Almost all SQL servers have a command line interface. The interface will vary from database to database, but an example with Sqlite would be:

(sqlite3 someuser.db "select fname,lname,age,job from users" ) | tr '|' '\n' | zenity --list \
  --title="My Database" \
  --column="first name" --column="last name" --column=age --column=job

Form Dialog – Insert SQL Data

The Forms Dialog allows for date, text and password inputs, and the result are passed as string (| is the default separator). A form example with output would be:

$ row=$(zenity --forms --title="Create user" --text="Add new user" \
   --add-entry="First Name" \    
   --add-entry="Last Name" \    
   --add-entry="Age" \    
   --add-entry="Job") ; echo $row

field1|field2|field3|field4

The next step is to format the form data into an SQL statement. The SQL INSERT syntax is:

INSERT INTO table (field1,field2…) VALUE (value1,value2…)

For the example above, field1|field2|field3|field4 needs to formatted to the values. This manipulation can be done by the bash sed command with search and replace (s) option:

$ row="field1|field2|field3|field4"
$ echo "'$row'" | sed "s/|/','/g"
'field1','field2','field3','field4'

The bash script to present the zenity form and input the data is below. The if statement is used to ensure that the cancel button wasn’t pressed. More if statement would probably be required for some data validation.

# zen_sqlin.sh - create a form to add a new user into a SQLite3 database
row=$(zenity --forms --title="Create user" --text="Add new user" \ --add-entry="First Name" \ --add-entry="Last Name" \ --add-entry="Age" \ --add-entry="Job") if [[ -n $row ]] # Some data found then indata=$(echo "'$row'" | sed "s/|/','/g") cmd="sqlite3 someuser.db \"INSERT INTO users (Fname,Lname,Age,Job) VALUES ($indata)\"" eval $cmd echo "Added data: $indata" fi

The script is run by: bash zen_sqlin.sh

Zenity (GTk) Warning Messages

Depending on your system you might see some Zenity warning messages such as:

Gtk-Message: 15:30:52.461: GtkDialog mapped without a transient parent. This is discouraged.

I never saw this on my Raspberry Pi but I did see it on my lubuntu system. To make things cleaner the warning can be piped to the null device:

$ zenity --info --text=$(date +'%S' )   --title="Seconds Timer Test" 2>/dev/null

Some Final Comments

I really just touched the surface on what zenity can do. For more info see some of the tutorials.

For simple stuff zenity works fine. If you’re looking for a more complete command line GUI tool try YAD, for myself I’ll stick to Python.

As a side note, there is a Python library for zenity. If you’re feeling comfortable with the bash version of zenity and you only need to do simple dialogs then this might be a good fit.

Using AWK in Bash Scripts

In the past Python has been my go to language for quick scripts, however lately I’ve done a lot of projects where I’ve needed to use small Bash scripts.

I found that by adding a little bit of AWK to my Bash scripts I’ve been able to do something in one line of Bash/AWK that would of taken me multiple lines of Python.

AWK is named after it’s authors: Alfred Aho, Peter Weinberger, and Brian Kernighan, and it is an old school (1994) text extraction and reporting tool.

The nice thing about AWK is that you only need to learn a couple of commands to make it usefully.

Get a Specific Row and Column Item

The iostat command can be used to show CPU stats. To get the idle time the 6th item in the 4th needs to accessed:

The AWK code can read the piped information and the AWK NR (row number) variable can be used filter just that row. The CPU idle time is item 6 (variable $6)

~$ iostat | awk '{if (NR==4) print $6}'
96.92

AWK logic need to be in single quotes and curly brackets groups together statements. This logic says: if the Number of Record (NR) variable is 4 print the 6th item.

Integer and Float Math, Variables and Formatting

Managing integers and floats in Bash can be a little challenging, luckily awk can offer some help.

Below is an example of float math and printing (yes… using a let with bc might be easier… but this is an example):

$ # Do math with printf
$ echo "3 4" | awk '{a=$1; b=$2; printf "%0.2f \n", (a / b) }' 
0.75
 
$ # Do math in awk and format the print (4 decimals)
$ echo "3 4" | awk '{c=$2/$1; printf "%0.4f \n", c }' 
1.3333 

$ # Send awk output to a variable
$ d=$(echo "3 4" | awk '{c=$2/$1; printf "%0.4f\n", c }') 
$ echo $d
1.3333

Below is an example of getting just the CPU temperature from the sensors utility and stripping out the “+” and “°C”:

$ sensors
dell_smm-virtual-0
Adapter: Virtual device
Processor Fan: 2706 RPM
CPU:            +44.0°C  
Ambient:        +37.0°C  
SODIMM:         +36.0°C  

$ sensors | grep CPU
CPU:            +44.0°C  

$ sensors | grep CPU | awk '{printf "%d\n", $2}'
44

Formatting Text File Output

To get specific columns, print them as required. The example below only prints columns 1 and 3:

$ cat pi_data.txt
time temp wave(ft) comments
---- ---- -------- --------
10:00 24   3       No wind
12:00 26   5       High winds
14:00 25   4       wind calming down

$ # print columns 1 and 3 with a tab between
$ cat pi_data.txt | awk '{print $1 "\t" $3}'
time	wave(ft)
----	--------
10:00	3
12:00	5
14:00	4

Output can be filter based on an item in a row. For example only print if the 1st item is a number:

$ cat pi_data.txt | awk '{if ( $1 ~ /[0-9]/ ) print $0}'
10:00 24   3       No wind
12:00 26   5       High winds
14:00 25   4       wind calming down

If-Else Logic

More complex logic can be added with if-else logic. Below is an example that pipes only the data and then it changes a column value to a string based on a condition:

$ cat pi_data.txt
time temp wave(ft) comments
---- ---- -------- --------
10:00 24   3       No wind
12:00 26   5       High winds
14:00 25   4       wind calming down

$ # Show time and small or medium for wave size
$ cat pi_data.txt | \
>   awk '{if ( $1 ~ /[0-9]/ ) print $0'} | \
>   awk '{if ($3 < 4) {print $1 "\t small"} else { print $1 "\t medium"} }'
10:00	 small
12:00	 medium
14:00	 medium

A single AWK command to adjust the title and then change the data:

$ cat pi_data.txt | \
>   awk '{if ( $1 ~ /[0-9]/ ) \
>            { \
>               {if ($3 < 4) {print $1 "\t small"} else { print $1 "\t medium"} } \
>        } else { print $1 "\t " $3} \
>        }'
time	 wave(ft)
----	 --------
10:00	 small
12:00	 medium
14:00	 medium

Math on a Column of Data

Bash is easy to use for a row of data, but I find it tricky on columns of data.

As an example, to get the total size of all Sqlite files, ls – l *db is piped to an awk statement. This awk statement that has two parts the first part sums column five (sum +=$5), the END is used to do line-by-line, the second part prints the result.

pete@lubuntu:~/dbs$ ls -l *.db 
-rw-r--r-- 1 pete pete  323584 Apr 14  2020 ebola.db
-rw-r--r-- 1 pete pete 5124096 Apr 14  2020 netflix.db
-rw-r--r-- 1 pete pete   98304 Apr  8  2020 sars.db
-rw-r--r-- 1 pete pete  503808 Apr 14  2020 schools.db
-rw-r--r-- 1 pete pete  208896 Apr  8  2020 schools_old.db
-rw-r--r-- 1 pete pete    8192 Feb  8 20:11 someuser.db
pete@lubuntu:~/dbs$ ls -l *.db | awk '{sum +=$5 } END {print "Total= " sum}'
Total= 6266880

I used this approach to find how much power is being consumed on all my USB ports:

pete@lubuntu:~/dbs$ lsusb -v  2>&- | grep -E  'Bus 00|MaxPower'
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
    MaxPower                0mA
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    MaxPower                0mA
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
    MaxPower                0mA
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    MaxPower                0mA
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    MaxPower                0mA
Bus 003 Device 004: ID 413d:2107  
    MaxPower              100mA
Bus 003 Device 003: ID 04b3:310c IBM Corp. Wheel Mouse
    MaxPower              100mA
Bus 003 Device 002: ID 1a40:0101 Terminus Technology Inc. Hub
    MaxPower              100mA
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    MaxPower                0mA

pete@lubuntu
:~/dbs$ (lsusb -v 2>&- | grep MaxPower | grep -o -E '[0-9]+' ) | awk '{ sum += $1} END {print "\nTotal= " sum " mA"}' Total= 300 mA

Finding and Killing a Task

There are a number different approaches to doing. To find and kill a number of task:

#!/bin/bash
#
# stop_task.sh - stop a task
#
task1="edublocks"

echo "Stopping $task1..."
ps -e | grep -E $task1 | \
 awk '{print $1}' | xargs sudo kill -9 1>&-

Some Useful AWK Statements

There is an enhanced version of AWK, GAWK (GNU AWK) that might already be loaded on your Linux system. If you are on a Raspberry Pi you can install GAWK by:

sudo apt-get install gawk

There are some excellent tutorials on AWK below are some of commands that I’ve found useful:

substr(string,position,length) – get part of a string:

An example of substr could be used to get the CPU temperature from the sensors utility:

~$ sensors | grep CPU | awk '{print substr($2,2,4)}'
 44.0

The substr() command looks at the 2nd item (+44.0°C), and starts at the 2nd character and it gets 4 characters.

print() with if() – print based on conditions:

The AWK print statement can be used with an if statement to show a filtered list.

An example of this would be to filter the ps (snapshot of the current processes) command, and print only lines with a time showing:

~$ # SHOW ALL PROCESSES
~$ ps -e 
   PID TTY          TIME CMD
     1 ?        00:00:03 systemd
     2 ?        00:00:00 kthreadd
     4 ?        00:00:00 kworker/0:0H
     6 ?        00:00:00 mm_percpu_wq
     7 ?        00:00:00 ksoftirqd/0
     8 ?        00:01:10 rcu_sched
...
~$  # SHOW ONLY PROCESSES WITH TIME
~$ ps -e | awk '{if ($3 != "00:00:00") print $0}'
   PID TTY          TIME CMD
     1 ?        00:00:03 systemd
     8 ?        00:01:10 rcu_sched
    10 ?        00:00:06 migration/0
    15 ?        00:00:03 migration/1
...

systime() / strftime() – get time and format time:

These time functions allow you to add time stamps and then do formatting on the date/time string. I found this useful in logging and charting projects. An example to add a time stamp to the sensor’s CPU temperature would be:

$ sensors | grep CPU | awk '{print strftime("%H:%M:%S ",systime()) $1 $2 }'
 11:06:18 CPU:+45.0°C

Final Comments

I’ve found that learning a little bit of AWK has really paid off.

AWK supports a lot of functionality and it can be used to create full on scripting applications with user inputs, file I/O, math functions and shell commands, but despite all this I’ll stick to Python if things get complex.