Category: Uncategorized

Home automation packages offer a huge selection of data import and integration tools, but they can never cover everything.

Many of us have some personal or hobby topics that we check periodically. These personal data points could be the wave height at a local surf spot, the bug levels at a favorite camping area, or when is the best time of go fishing.

In this blog I will look at how to scrape some hobby data points from web pages with just a single line of Bash code. I will then show how these results can be integrated into two home Internet of Things packages, Home Assistant and Node-Red.

Getting Started

Each of the different automation solutions offer web scraping tools. For example Python has the Beautiful Soup library, Home Assistant has the Scrape sensor, and Node-Red has the Scrape-it flow. These web scraping tools are all very usable, but unfortunately they require a detailed knowledge of the HTML/Document-Object-Model for the requested page.

A simple alternate approach is to use the lynx text based browser. This command line tool can be used to strip out all the HTML tagging and dump out just the text on a page. The output from lynx can be piped to common commands like grep, sed, and awk to filter out the required values. The advantage in the lynx approach is that you don’t need to understand the internal HTML on a page, and it only takes 1 line of Bash to get what you need.

To install Lynx on Raspian/Debian/Ubuntu:

sudo apt install lynx

Two Off-line Examples

The first step in developing custom web scraped data points is to find the required Bash commands. Working directly on either Home Assistant or Node-Red can be a little challenging. Luckily, you can do all your basic testing on a laptop and then once you’ve got things working you can move the code over to your IoT system.

Two personal interest topics for me, are the pollen levels in a local hiking region, and the amount of snow at a ski resort that I’m planning to take the family to.

The Lynx -dump option will output a stream of text with HTML tags, HTML encoding and Javascript removed. The command syntax that I’m looking for is:

lynx -dump http://somepage.com  | "filter the output until I get the result"

The picture below shows how I used lynx to find the pollen level from a weather page. For this example I first looked at the top 10 lines in the output, and I compared this output to the actual web page. A quick check showed that the pollen level value was on the sixth line. The sed utility can be used to delete all but the sixth line by setting the option to : ‘6!d’.

The full Bash script to get the pollen level from my favourite hiking area is:

# define the weather page URL
theurl="https://www.theweathernetwork.com/en/city/ca/ontario/lions-head/pollen"
# get the pollen value (on the 6th line)
lynx -dump $theurl | sed '6!d'

The second example, (picture below), uses lynx with a grep call to find the text “Top Lift” on a web page. In this snapshot, the output is returned as four words: Top Lift: 2.3 m. The snow depth (2.3) is the third word in the string. There are a few ways to extract words in a string, and for this example I used the awk utility.

The full Bash script to get the snow base for my upcoming ski trip is:

# define the ski resort URL
theurl="https://www.snow-forecast.com/resorts/Whistler-Blackcomb/6day/mid"
# find the line with "Top Lift", and 
# then parse out the 3rd (snow) value
lynx -dump $theurl | grep 'Top Lift' | awk '{ print $3 }'

Now that I’ve got the Bash commands for my personal data points I can move to the next step of adding them to my Home Assistant or Node-Red systems.

Home Assistant Command Line Sensors

The Home Assistant (HA) Command Line sensors offer an interface that allows the output from Bash commands to be used as HA viewable entities.

Before our earlier commands can be run, the Lynx utility needs to be installed. There are several ways to install applications into HA. The important point is that Lynx needs to be available in the same working space that the Command Line sensors run in. A simple way to ensure this, is to use a Command Line sensor to install Lynx directly.

To install sensors, modify the /config/configuration.yaml file. This file can accessed through the File Editor or Terminal HA add-ons, or via a secure shell (SSH) connection.

The picture below shows a temporary sensor called Install_Lynx that has been added to /config/configuration.yaml. This sensor will run the apk add command to install software. After this file is updated and saved, a Home Assistant restart will be required.

Once the required software is installed, it would be recommended that you remove this temporary “install” sensor, otherwise the system will try to re-install Lynx every 60 seconds.

Another approach would be to only install the software if isn’t there. (Note: the sensor is still run every minute by default). The command for this would be:

if ! ( apk list lynx ); then apk add lynx; fi

After Lynx is installed some new command line sensors can be added that access the personal web page data. The image below shows a sample /config/configuration.yaml file with the Install_Lynx sensor removed and two new sensors added that use our Bash commands.

These web pages don’t update too frequently so the scan_interval is set to 1 hour (3600 seconds). Also it good to ensure that the Lynx command is given enough time to run, so I set the command_timeout to 15 seconds. Like in the previous step HA needs to be restarted after new sensors are added.

The final step is to put the new sensor tags into a viewable presentation. The Home Dashboard can be selected from HA’s Overview option. The pic below shows the addition of a card that contains the newly created web scraped command line sensors.

Using Lynx with Node-Red

Like Home Assistant, the Lynx utility needs to be loaded on the Node-Red system. For Raspian/Debian/Ubuntu system this is done by: sudo apt install lynx.

To test that Lynx is working within Node-Red an inject, exec and a debug node can be used.

For my test I used the Bash statement to find pollen levels as the command in the exec flow. It’s important to note that the top output connector of exec flow is the output result (stdout).

The next step is to create some logic that schedules the scraping of web pages and then shows the data on a web dashboard. The image below shows a Node-Red example that uses a Big Timer to trigger the scraping of web pages. For the presentation of data a Text dashboard flow is used to show the pollen levels, and a Gauge dashboard flow is used for the snow levels.

The realtime data is available at: http://my_node_red_ip:1880/ui .

Summary

To create a successful home automation solution it’s important to access and view all the relevant signals.

From my personal experience I’ve found that once I setup my key measuring and controlling devices I would only infrequently look at the system. However, since I started adding hobby topics to the system I’ve noticed that I’m using the system a lot more often.

While I was working on an older Puppy Linux system I came across qtkdialog.

I’m a huge fan of Zenity and Yad for making Bash dialog, but qtkdialog has the advantage in that it allows users to create custom interfaces using an XML-like syntax.

Unfortunately qtkdialog is fairly old, so there isn’t a lot of documentation.

This blog references some of my notes. Some of the key features in qtkdialog include:

• free form layouts
• Custom text fonts, colors and sizing
• Support for frames and tabs

Installation

To install use the following steps:

sudo apt install build-essential
wget https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/gtkdialog/gtkdialog-0.8.3.tar.gz

tar -zxvf gtkdialog-0.8.3.tar.gz
cd gtkdialog-0.8.3/

./configure
make
sudo make install

If you plan on using a lot of icons, I’d recommend getting an icon browser/viewer. (For this I use yad-icon-browser that is bundled with yad).

A First Example

A first example a dialog with static text and one active button would be:

#!/bin/bash 
#
# gtk1.sh - gthdialog example with text and two buttons
#
export MAIN_DIALOG=' 

<window title="My First Program" icon-name="gtk-about"> 
  <vbox>
    <text>	 
      <label>Static Text and one active button</label> 
    </text>
    <button>
      <label>Echo the date</label>
      <action>echo "$(date)"</action> 
     </button> 
     <button ok></button>
  </vbox> 
</window> 
' 
# Open the dialog 
gtkdialog --program=MAIN_DIALOG --center

For this example (gtk1.sh), a variable, MAIN_DIALOG, is created with all the XML code. This variable is then passed to gtkdialog using the –program option. An alternate approach would be to pass an XML file to the program:

gtkdialog --file=gtk1.xml

Some of the key points in this example are:

• start with <window> and end with </window>
• format the presentation with vertical (vbox) or horizontal (hox) tags (at least 1 is required)
• static labels must be within a <text> </text> block
• buttons can call Bash code using <action> tags

A Diagnostic Example

A more advanced example will use a couple of frames to group together buttons with icons. These buttons will launch a Zenity windows with diagnostic data.

#!/bin/bash 
#
# gtk5.sh - create a diagnostic interface
#

GTKDIALOG=gtkdialog 

# Pass a custom color file if required
#export GTK2_RC_FILES=gtkrc_mono

export MAIN_DIALOG=' 

<window 
   title="Diagnostics" 
   icon-name="applications-utilities"
   name="MyWindowBg" >
  <vbox>
    <text use-markup="true"><label>"
      <span color='"'red'"' font-family='"'Comic'"' weight='"'bold'"' size='"'xx-large'"'>
        <i>System Tools</i>
      </span>"</label>
    </text>

    <hbox>
      <frame CPU Info>	
	<hbox>
	  <button>	    
            <input file icon="view-sort-descending"></input>
            <label>TOP - Table Of Processes</label>
            <width>30</width>
	    <action>top -b -n 1 | zenity --text-info --title "TOP"</action>
	  </button>  
	</hbox>
	<hbox>
	  <button use-underline="true" tooltip-text="Run VMSTAT Utility">
	    <input file icon="gnome-dev-memory"></input>
            <label>VMSTAT - CPU Stats </label>
            <width>55</width>
	    <action>yad --text="$(sensors)" --title=Sensors</action>
	  </button>  
	</hbox>
      </frame>

      <frame Device Info>	
	<hbox>
	  <button>	    
            <input file icon="media-removable-symbolic"></input>
            <label>LSUSB - List USB Devices</label>
            <width>40</width>
	    <action>lsusb | zenity --text-info --title "lsusb"</action>
	  </button>  
	</hbox>
	<hbox>
	  <button>
	    <input file icon="harddrive"></input>
            <label>LSPCI - List PCI Devices </label>
            <width>45</width>
	    <action>lspci | zenity --text-info --title "lspci"</action>
	  </button>  
	</hbox>
      </frame>

    </hbox>
    <button ok></button>
  </vbox>
</window> 
' 

case $1 in 
	-d | --dump) echo "$MAIN_DIALOG" ;; 
	*) $GTKDIALOG --program=MAIN_DIALOG --center ;; 

esac

Some of the key points in the code are:

• Font color, type and size can be adjusted with the <span> tag. Note: see the Pango Markup Language for details.
• To put icon on button use: <input file icon=xxxx></input>
• Output from Bash commands can be piped to the Zenity dialog tool.

Custom Colors

A color file can be used to define many of the gtkdialog objects. To reference a custom color file:

• Add a name item to the window object (< window name=”MyWindowBg”)
• Create a GTK2_RC_FILES variable and reference the color file ( export GTK2_RC_FILES=gtkrc_mono )

The color file (gtkrc_mono in this example) references the name of the window (MyWindowBg) and any required custom color settings.

style "bgWhite" { bg[NORMAL] = "#FFFFFF" }
style "fgWhite" { fg[NORMAL] = "#FFFFFF" }
style "bgBlack" { bg[NORMAL] = "#000000" }
style "bgRed" { bg[NORMAL] = "#FF0000" }
style "fgRed" { fg[NORMAL] = "#FF0000" }
style "bgGreen" { bg[NORMAL] = "#FFFF00" }
style "fgGreen" { fg[NORMAL] = "#FFFF00" }
style "bgBlue" { bg[NORMAL] = "#0000FF" }
style "fgBlue" { fg[NORMAL] = "#0000FF" }

widget "MyWindowBg" style "bgBlack"
widget "MyWindowBg.GtkVBox.GtkHBox.MyButtonBg" style "bgGreen"

widget "MyWindowBg.GtkVBox.myEVB" style "bgBlue"

widget_class "*<GtkFrame>.GtkLabel" style "fgWhite"
widget_class "*<GtkButton>.*.GtkLabel" style "fgBlue"

For the earlier example if the line : # export GTK2_RC_FILES=gtkrc_mono is changed to:

# Pass a custom color file if required
export GTK2_RC_FILES=gtkrc_mono

The dialog will be changed to a black background with white frames and blue button text:

Summary

Understanding gtkdialog is a useful exercise, especially if you’re working on older system.

Manually creating gtkdialog layouts can quickly become painful. Tools like Glade offer a visual interface for creating complex graphic interfaces.

There are some Python libraries that support gtk graphic interfaces.

I’ll probably stick using basic Zenity or Yad for my Bash GUIs, or Python for more complex projects.

In the past Python with the Beautiful Soup library has been a great approach for my web scraping.

I was recently doing a small project and I was amazing at what one Bash statement would get me.

My approach was to use the text based Lynx browser and pipe the output to a grep search.

Below is an example where I used Lynx to dump the “Sunshine Village Snow Forecast” web page to find how much snow they had.

The Lynx Text Browser

The first step in web scraping is to get a web page into a searchable format.

I started out by looking at using cURL with the html2text tool, but I found that using the Lynx browser offered a one step solution with a cleaner text output.

To install Lynx on Raspian/Debian/Ubuntu use:

sudo apt install lynx

The lynx -dump option will output a web page to text with HTML tags and Javascript removed. It’s important to note that what you see on the page may not match the outputted text.

Below is an example where I wanted to get the new snow at Sunshine Village. On the web page Javascript is used to show the snow depth as either centimetres or inches, but on the text output both units and their values are shown.

Bash has a good selection of string manipulation tools. Below is an example to extract the first part of string to only show the snow in centimeters (cm):

$ theurl="https://www.snow-forecast.com/resorts/Sunshine/6day/mid"
$ thestr="New snow in Sunshine Village:"
$ # Create a variable with the result string from Lynx
$ newsnow=$(lynx -dump "$theurl" | grep "$thestr")
$ 
$ echo "$newsnow"
   New snow in Sunshine Village:  4.8cm1.9in on Fri 8th (after 3 PM)
$    
$ # Get the first part of the string before "cm"
$ # The %% gets the first part
$ echo "${newsnow%%cm*} cm"
   New snow in Sunshine Village:  4.8 cm

My Final App

We were going on a family ski trip and to get pumped I created a morning notification script that showed the new morning snow and the base.

#!/bin/bash
#
# skitrip.sh - show the Sunshine ski conditions in a notification
#
theurl="https://www.snow-forecast.com/resorts/Sunshine/6day/mid"

# Get the new snow depth
thestr="New snow in Sunshine Village:"
result=$(lynx -dump "$theurl" | grep "$thestr")
newsnow="${result%%cm*} cm"

# Get the base
thestr="Top Lift:"
base=$(lynx -dump "$theurl" | grep "$thestr")

# Show the results in a desktop notification
msg="$newsnow\n$base (base)"
icon="$HOME/Downloads/mountain.png"
notify-send -t 10000000 -i "$icon"  "Sunshine Ski Resort" "$msg"

The notify-send utility will put a message on a Linux desktop, another option could be to send a SMS message.

Summary

Scraping web pages can be tricky and the pages can change at anytime.

I found that Lynx worked on many pages but not all.

The grep utility is extremely useful and it offers a lot of interesting options, such as getting lines before or after the found string.