Automate Web Page Logins

To ensure a safe number of users during Covid we had to log into Web sites for access to activities like pools, gyms and ski hills. These precautions made sense, however the booking process was awkward and was easy to miss an activity if you weren’t signed up early enough.

Luckily there are some great Linux tools that can be used to automate web logins. If you are working in MS-Windows take a look at SendKeys command.

Linux Mouse and Key Stroke Simulation

In the Linux environment there a number of choices, such as xautomation and xdotools.

I thought that the xdotools package was feature rich with a number of special functions for desktop and windows functions. The xautomation package was a little simpler and it has some Python wrappers.

I found that it was very useful to install wmctrl, this allows you to easily see which windows are running and it can set the active window with a substring (you don’t need the full window name).

To install xautomation and wmctrl in Ubuntu:

sudo apt-get install xautomation wmctrl

Log in Example

To create an automation script you need to do the required steps once manually and document what is a Tab, entry field or an Enter/Return key. Some trial and error work will probably be required for timing if you are working between pages links.

A good simple example is to try and log into Netflix.

The following Bash script will : 1) open a Chrome browser page, 2) set the focus to the page and 3) sent the correct tab, text and a return key.


#!/bin/bash
# netflix_login.sh - script logs into Netflix
#

url="https://www.netflix.com/ca/login"
email="your_email.com"
pwd="your_password"

chromium-browser $url & #open browser to

# wait for the page to open, the set focus to it
sleep 2
wmctrl -a "Netflix - Chromium"
sleep 1 # allow time to get focus before sending keys
xte "key Tab"
xte "key Tab"
xte "str $email"
xte "key Tab"
xte "str $pwd"
xte "key Tab"
xte "key Tab"
xte "key Return"

echo "Netflix Login Done..."

Use wmctrl -l to see which windows are open.

A specific window can have the focus based on a substring, with the -a options

The xte command uses a string to pass key, text and mouse actions to the active window.

Final Comments

Using simulated keyboard and mouse movements is a nice tool for automating simple web pages.

If you need to look at more complex web automation solutions check out Selenium, it has a Python library that allow you to access the DOM object of the open web browser page.

Orange Pi – $2-$50 Raspberry Pi Competitor

Orange Pi is a low-cost Raspberry Pi competitor. It is developed by the Chinese Shenzhen Xunlong CO Software company. Just like the Raspberry Pi, Orange Pi is an Open Source project.

Orange Pi offers a wide variety of offerings, that start at $2 and go up from there.

I purchased an Orange Pi Lite for about $15, and I’ve been quite happy with it.

Compared to the Raspberry Pi, Orange Pi is:

  • less expensive. Faster for the price
  • has more hardware options and form factors

This blog documents some of my installations steps to get an Orange Pi (Lite) running with Raspberry Pi functionality.

Some Interesting OrangePi Offers

There are quite a few different modules that are available. Some of the ones that I found interesting are:

Orange Pi Zero – there a few model, the 256MB LTS ($2), to the Zero 2 ($30).

Orange Pi 3G-IOT-B – includes a built-in 3G SIM card support ($22). There are also 4G models.

Base Installation

Like the Rasp Pi there are a number of OS options that can be installed. I chose the Armbian OS, which seems to be the most popular.

There are a number of ways to put an image on an SD chip, I like: pi imager. Once an image has been installed, the command line can be used to get the basic things setup and installed

  ___  ____  _   _     _ _       
 / _ \|  _ \(_) | |   (_) |_ ___ 
| | | | |_) | | | |   | | __/ _ \
| |_| |  __/| | | |___| | ||  __/
 \___/|_|   |_| |_____|_|\__\___|
                                 
Welcome to Armbian buster with Linux 5.4.43-sunxi

System load:   0.00 0.00 0.00  	Up time:       21 min		
Memory usage:  41 % of 492MB  	IP:            192.168.0.113
CPU temp:      52°C           	
Usage of /:    16% of 15G    	

Last login: Sat Apr 17 12:28:07 2021 from 192.168.0.111

On Pi the config tool is called raspi-config, on the Ambian OS it’s : armbian-config

This tool is used to setup initial key things like: Wifi, Bluetooth, SSH, Desktop setting etc.

Depending on the OS that has been installed you may have everything you need or you may need to do further installs. For myself I needed to load “Pi type” features such as: the gpio utility, Python GPIO library and Node-Red.

I logged in as root for all my base installations.

GPIO (WiringPi) Utility

The gpio utility is super useful for manual seeing and setting GPIO pins.

To install gpio :

git clone https://github.com/orangepi-xunlong/WiringOP
cd WiringOP
chmod +x ./build
sudo ./build

The pinouts are different that the Rasp Pi, to see them enter: gpio readall

root@orangepilite:~# gpio readall
 +------+-----+----------+------+---+OrangePiH3+---+------+----------+-----+------+
 | GPIO | wPi |   Name   | Mode | V | Physical | V | Mode | Name     | wPi | GPIO |
 +------+-----+----------+------+---+----++----+---+------+----------+-----+------+
 |      |     |     3.3V |      |   |  1 || 2  |   |      | 5V       |     |      |
 |   12 |   0 |    SDA.0 |  OFF | 0 |  3 || 4  |   |      | 5V       |     |      |
 |   11 |   1 |    SCL.0 |  OFF | 0 |  5 || 6  |   |      | GND      |     |      |
 |    6 |   2 |      PA6 |  OFF | 0 |  7 || 8  | 0 | OFF  | TXD.3    | 3   | 13   |
 |      |     |      GND |      |   |  9 || 10 | 0 | OFF  | RXD.3    | 4   | 14   |
 |    1 |   5 |    RXD.2 |  OFF | 0 | 11 || 12 | 0 | OFF  | PD14     | 6   | 110  |
 |    0 |   7 |    TXD.2 |  OFF | 0 | 13 || 14 |   |      | GND      |     |      |
 |    3 |   8 |    CTS.2 |  OFF | 0 | 15 || 16 | 0 | OFF  | PC04     | 9   | 68   |
 |      |     |     3.3V |      |   | 17 || 18 | 0 | OFF  | PC07     | 10  | 71   |
 |   64 |  11 |   MOSI.0 |  OFF | 0 | 19 || 20 |   |      | GND      |     |      |
 |   65 |  12 |   MISO.0 |  OFF | 0 | 21 || 22 | 0 | OFF  | RTS.2    | 13  | 2    |
 |   66 |  14 |   SCLK.0 |  OFF | 0 | 23 || 24 | 0 | OFF  | CE.0     | 15  | 67   |
 |      |     |      GND |      |   | 25 || 26 | 0 | OFF  | PA21     | 16  | 21   |
 |   19 |  17 |    SDA.1 |  OFF | 0 | 27 || 28 | 0 | OFF  | SCL.1    | 18  | 18   |
 |    7 |  19 |     PA07 |  OFF | 0 | 29 || 30 |   |      | GND      |     |      |
 |    8 |  20 |     PA08 |  OFF | 0 | 31 || 32 | 0 | OFF  | RTS.1    | 21  | 200  |
 |    9 |  22 |     PA09 |  OFF | 0 | 33 || 34 |   |      | GND      |     |      |
 |   10 |  23 |     PA10 |  OFF | 0 | 35 || 36 | 0 | OFF  | CTS.1    | 24  | 201  |
 |   20 |  25 |     PA20 |  OFF | 0 | 37 || 38 | 0 | OFF  | TXD.1    | 26  | 198  |
 |      |     |      GND |      |   | 39 || 40 | 0 | OFF  | RXD.1    | 27  | 199  |
 +------+-----+----------+------+---+----++----+---+------+----------+-----+------+
 | GPIO | wPi |   Name   | Mode | V | Physical | V | Mode | Name     | wPi | GPIO |
 +------+-----+----------+------+---+OrangePiH3+---+------+----------+-----+------+

Python GPIO Library

The Raspberry Pi Python GPIO library as been ported to the Orange Pi and it’s called OPi.GPIO. To install it:

# if required install PIP for Python 2 and 3
apt install python-pip python3-pip

# install Orange Pi GPIO library for Python 2 and 3
pip install OPi.GPIO
pip3 install OPi.GPIO

A quick test from to test that things are working:

root@orangepilite:~# python3
Python 3.7.3 (default, Dec 20 2019, 18:57:59) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import OPi.GPIO as GPIO
>>> GPIO.setmode(GPIO.BOARD)
>>> GPIO.setup(12, GPIO.OUT)
>>> # set pin 12
... 
>>> GPIO.output(12, 1)
>>> GPIO.input(12)
1
>>> GPIO.cleanup()

Node-Red

To install Node-Red, (logged in as root), use the following command, and follow the prompts (don’t install Raspberry support). This install takes about 20 minutes and Node.js will also be added:

bash <(curl -sL https://raw.githubusercontent.com/node-red/linux-installers/master/deb/update-nodejs-and-nodered)

The next step is to install the Orange PI GPIO support:

cd $HOME/.node-red
npm install node-red-contrib-opi-gpio

To manual start Node-Red enter: node-red-start & . To manually stop Node-Red: node-red-stop. See the Node-Red documentation for more info.

A simple circuit to set and read GPIO would be:

Final Comments

A Orange Pi is awesome for generic or simple hardware projects.

If however you’re doing projects where you need tops, for motor controls or relays then I would still to a Raspberry Pi.

Re-purpose an eReader

My daughter and I looked at trying to re-purpose an old eReader. Our goal was to make a kitchen kiosk display that would show us items of daily interest. For us that included news, stocks and weather data that we pulled from the Internet, along with some local data from Arduino, Raspberry Pi and a Home Assistant nodes.

In this blog we’ll look at:

  • How to install a different OS on the eReader
  • Using Linux on an eReader
  • Python eReader considerations
  • Kiosk mode Web Browser pages

Opening up the eReader

For our project we used a Kobo mini, so the procedure to load a new OS will vary somewhat based on the eReader manufacturer.

Once you open up the eReader, you’ll have access to an microSD inside the unit.

I would highly recommend keeping the original SD in case that you ever need to roll back to the original setup.

For installation instructions on how to load different operating system for Kobo eReaders see: https://sites.google.com/site/gibekm/hardware/kobo/kobo-as-tablet. The two main OS choices are Android or Linux. For our project we felt that Linux would be a cleaner option. Images can be downloaded from: https://www.dropbox.com/sh/snsdg1c5cg21kws/3LfelXgbGe.

If you are trying to re-purpose other brands of eReaders there appears to be lots of how to guides, for example for Kindle see: https://www.lifehacker.com.au/2016/07/how-to-jailbreak-your-kindle .

Installing Debian Linux

Once you’ve downloaded your required OS you’ll need to put in on a micro-SD chip.

There are a number of different tools to move and copy images, because I’m a Raspberry Pi user I like to use the Raspberry Pi Imager (rpi-imager). The rpi-imager utility runs on Linux, Windows and Mac OS. To install it on Linux:

sudo snap install rpi-imager

Using rpi-imager, select the “use custom” option.

Depending on your eReader and the OS you might be ready to go. Unfortunately for our Kobo mini installation we needed to another step. The added step that we needed to do moved some files from the original SD chip on to our new SD chip. This added step gave us a clean double-boot install so we could either run the original Kobo software or boot into Debian Linux. Our added steps were:

# insert the original microsd
sudo dd if=/dev/mmcblk0 bs=512 skip=1024 count=1 of=/home/you/path/to/image/original.img
# insert the new microsd with the debian image on it
sudo dd if=/home/ian/Desktop/kobo/original.img bs=512 seek=1024 count=1 of=/dev/mmcblk0

To setup the Wifi we needed to boot into the Kobo system and configure our network settings. Once that was complete we were able to have Wifi networking on the Linux side.

Linux on the eReader

The first step is to enable the Wifi, this option should be on the main menus.

eReaders have great battery life when they are used as an eReader, however when they are 100% on Wifi their battery life will be more like a standard tablet, with 5-6 hours of life.

The eReader has a floating keyboard so you can do some work directly on the unit, but doing an ssh connection from a PC is definitely easier.

The menuing on the eReader will vary based on the OS that is loaded. On our Kobo, the Debian OS used the awesome window manager. To add items into the menus look for the file: .config/awesome/rc.lua . We added two extra entries, one for a Python app and the second for a Browser kiosk app :

-- This is in the /awesome/rc.lua menu file
-- ...

menuapps = {
   { "MyApp","/home/marek/temp/g4.py"},
   { "MyWebpage","/home/marek/temp/mywebpage.sh"},
   { "Firefox", "firefox" },
   { "FBReader", "fbreader" },
   { "Calculator", "xcalc" }
}

Python on an eReader

For our first Python test app we wanted to :

  • 100% fill the eReader screen
  • Find different Font sizes that worked well on an eReader
  • Have a large “Close” button.
# Simple Tkinter Clock
#
from Tkinter import *
import time

def update_clock():
        now = time.strftime("%H:%M")
        lbl_time.configure(text=now)
        today = time.strftime("%A %B %d")
        lbl_date.configure(text=today)
        window.update()
        window.after(5000, update_clock)

window=Tk()

lbl_time=Label(window, text="", font=("Helvetica", 128))
lbl_time.pack()
lbl_date=Label(window, text="", font=("Helvetica", 64))
lbl_date.pack()
btn=Button(window, text="Close",font=("Helvetica", 32),bg='grey', command=window.destroy)
btn.pack()

window.title('Kitchen Kiosk')
window.geometry("800x600+0+0")
window.attributes("-fullscreen", True)
window.after(1000, update_clock)
window.mainloop()

We found that it took a bit of time to play with font sizing and gray tones before we had something that we liked. Our final Python app used weather data from our Home Assistant Node and we built some custom gauges.

Kiosk Browser App

Our second app showed a custom web page that collected data from a Raspberry Pi and our Home Assistant node.

The plan was to show the data on the web browser in full screen or kiosk mode on the eReader.

To run Firefox in kiosk mode:

firefox --kiosk http://mysite/thepage.htm

On our eReader the browser was iceweasel, which is a lighter weight browser which unfortunately does not support kiosk mode. A workaround for browsers that don’t support kiosk mode is to use xdotool which allows you to simulate mouse and keyboard actions. We wrote a script that opened iceweasel and then re-positioned the window:

#!/bin/bash
# mywebpage.sh - open web browers to our page and go full screen

iceweasel http://192.168.0.111:8080/ &
sleep 15
xdotool key F11

Final Comments

The Debian OS that we loaded on our eReader was fairly old and it didn’t support Python 3.7 or Firefox, but we found alternatives and we still could make it work.

If your applications doesn’t need to be constantly updated, for example only check weather and stocks every 15 minutes or so, then you could turn on the Wifi, get the data, then turn off the Wifi. This could greatly increase the battery life on the eReader.

Home Assistant (REST) API

There are a few methods to communicate with Home Assistant. In this blog I wanted to document my notes on using the REST API.

Getting Started

I found that loading the File Editor Add-on made configuration changes quite easy. To load the File Editor, select the Supervisor item, then Add-on Store:

With the File Editor option you be able to modify your /config/configuration.yaml file. For this you’ll need to add an api: statement. I also added a command line sensor that shows the Raspberry Pi CPU idle time. I did this so that I could see a dynamic analog value:

After I made these changes I restarted my HA application, by the “Configuration” -> “Server Controls”.

Next I needed to create a user token. Select your user and then click on “Create Token” in the Long-Lived Access Tokens section. This token is very long and it only is shown once, so copy and paste it somewhere save.

Access the REST API with CURL

Curl is a command line utility that exists on Linux, Mac OS and Windows. I work in Linux mostly and it’s pre-installed with Ubuntu. If you’re working in Windows you’ll need to install CURL.

Getting started with curl isn’t required and you got straight to programming in your favourite language, however I found that it was usefully testing things out in curl before I did any programming.

The REST API is essentially an HTTP URL with some headers and parameters passed to it. For a full definition see the HA API document. The key items in REST API are:

  • Request type – GET or POST (note: there are other types)
  • Authorization – this is where the user token is passed
  • Data – is used for setting and defining tags
  • URL – the Home Assistant URL and the end point (option to view or set)

To check that the HA API is running a curl GET command can used with the endpoint of /api/.

$ curl -X GET -H "Authorization: Bearer eyJ0eXAiO....zLjc"   http://192.168.0.103:8123/api/

{"message": "API running."}

The user token is super long so your can use the \ character to break up your command. For example:

curl -X GET \
   -H "Authorization: Bearer eyJ0eXAiOiJKV......zLjc" \
   http://192.168.0.106:8123/api/states

Read a Specific HA Item

To get a specific HA item you’ll need to know its entity id. This can found by looking at the “Configuration” -> “Entities” page:

For my example I created a sensor called Idle Time, its entity id is: sensor.idle_time.

A curl GET command with the endpoint of /states/sensor.idle_time will return information on this sensor. The full curl command and the results would look like:

$ curl -X GET   -H "Authorization: Bearer eyJ0eXAiOiJKV1Q....zLjc"  \   http://192.168.0.103:8123/api/states/sensor.idle_time

{"entity_id": "sensor.idle_time", "state": "98.45", "attributes": {"unit_of_measurement": "%", "friendly_name": "Idle Time"}, "last_changed": "2020-12-12T17:34:10.472304+00:00", "last_updated": "2020-12-12T17:34:10.472304+00:00", "context": {"id": "351548f602f5a3887ff09f26903712bc", "parent_id": null, "user_id": null}}

Write to a New HA Item

A new or dynamic items can be created and written to remotely using a POST command with the definitions included in the data section. An example to create an entity called myput1 with a value of 88.6 would be:

curl -X POST \
   -H "Authorization: Bearer eyJ0eXAiOi....zLjc" \
   -H "Content-Type: application/json" \
   -d '{"state":"88.6", "attributes": {"unit_of_measurement": "%", "friendly_name": "Remote Input 1"}}' \
   http://192.168.0.103:8123/api/states/sensor.myinput1

This new entity is now available to HA and shown on the dashboard.

Write to a Switch

If you have a writeable device such as a switch you can use the REST to remotely control it.

For myself I have a Wemo switch with an entity name of : switch.switch1.

To control the switch the entity id is passed in the data section and the endpoint uses either a turn_on or turn_off parameter.

curl -X POST \
   -H "Authorization: Bearer eyJ0eXAiOiJ....zLjc" \
   -H "Content-Type: application/json" \
   -d '{"entity_id": "switch.switch1"}' \
   http://192.168.0.103:8123/api/services/switch/turn_on

Python and the HA API

Python can parse the JSON responses from the reading a sensor value:

from requests import get
import json

url = "http://192.168.0.103:8123/api/states/sensor.idle_time"
token = "eyJ0eXAiOiJK...zLjc"

headers = {
    "Authorization": "Bearer " + token,
    "content-type": "application/json",
}

response = get(url, headers=headers)

print("Rest API Response\n")
print(response.text)

# Create a json variable
jdata = json.loads(response.text)

print("\nJSON values\n")
for i in jdata:
    print(i, jdata[i])

The output will be something like:

Rest API Response

 {"entity_id": "sensor.idle_time", "state": "98.46", "attributes": {"unit_of_measurement": "%", "friendly_name": "Idle Time"}, "last_changed": "2020-12-12T19:29:10.655530+00:00", "last_updated": "2020-12-12T19:29:10.655530+00:00", "context": {"id": "2509c01cadb9e5b0681fa22d914e7b10", "parent_id": null, "user_id": null}}

 JSON values

 entity_id sensor.idle_time
 state 98.46
 attributes {'unit_of_measurement': '%', 'friendly_name': 'Idle Time'}
 last_changed 2020-12-12T19:29:10.655530+00:00
 last_updated 2020-12-12T19:29:10.655530+00:00
 context {'id': '2509c01cadb9e5b0681fa22d914e7b10', 'parent_id': None, 'user_id': None}

To write a value to myinput1 in Home Assistant:

from requests import post
import json

url = "http://192.168.0.103:8123/api/states/sensor.myinput1"
token = "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJkMDI2YjAxY2VkZWU0M2E1OWY1NmI1OTM2OGU1NmI0OSIsImlhdCI6MTYwNzc5Mzc0NCwiZXhwIjoxOTIzMTUzNzQ0fQ.qEKVKdadxNWp249H3s_nmKyzQMIu5WDQkS9hiT-zLjc"

mydata = '{"state":"99.3", "attributes": {"unit_of_measurement": "%", "friendly_name": "Remote Input 1"}}'

headers = {
    "Authorization": "Bearer " + token,
    "content-type": "application/json",
}

response = post(url, headers=headers,data =mydata)

print("Rest API Response\n")
print(response.text)

# Create a json variable
jdata = json.loads(response.text)


print("\nJSON values\n")
for i in jdata:
    print(i, jdata[i])

The output will look something like:

Rest API Response

 {"entity_id": "sensor.myinput1", "state": "99.3", "attributes": {"unit_of_measurement": "%", "friendly_name": "Remote Input 1"}, "last_changed": "2020-12-12T20:40:05.797256+00:00", "last_updated": "2020-12-12T20:40:05.797256+00:00", "context": {"id": "31b422d02db41cde94470ebae7fac48c", "parent_id": null, "user_id": "1392a10c7bbb4cf0891a7f8a351740c7"}}

 JSON values

 entity_id sensor.myinput1
 state 99.3
 attributes {'unit_of_measurement': '%', 'friendly_name': 'Remote Input 1'}
 last_changed 2020-12-12T20:40:05.797256+00:00
 last_updated 2020-12-12T20:40:05.797256+00:00
 context {'id': '31b422d02db41cde94470ebae7fac48c', 'parent_id': None, 'user_id': '1392a10c7bbb4cf0891a7f8a351740c7'}

Final Comments

A REST API interface allows foreign devices such as PCs, Raspberry Pi and Arduino module to be used as remote I/O devices.

There are REST client HTTP libraries that are available for the Arduino, however it might be cleaner to implement an MQTT interface instead.

Make your Python Apps Run Faster

Python apps on lower end hardware like a Raspberry Pi can be a bit slow but luckily there are some options that you can do to improve things.

There are a number of interesting packages that allow you to compile, interpret or repackage your Python apps. In this blog I’d like to highlight two packages that I have had good success with:

  • Pypy – a replacement to the native Python interpreter. Your code can run more that 4 times faster !
  • Nuitka – a native Python utility to compile Python apps to C code !

Pypy – a faster Python

The Pypy site has some performance results, and they state that on average Pypy will run 4.2 times faster than native Python, for my test run Pypy ran 9 times faster than native Python.

The improved performance of Pypy is due to its just-in-time compiler, as opposed to the native Python’s line-by-line interpreter.

Guido van Rossum, creator of Python, has been even been quoted saying: “If you want your code to run faster, you should probably just use PyPy.”

Pypy has a Python 2.7 and 3.6 version that is available for Linux, MacOS and Microsoft Windows. To install the Pypy 3 version in Ubuntu :

sudo add-apt-repository ppa:pypy/ppa
sudo apt update
sudo apt install pypy3

Pypy3 can also be installed using snap, (this might be the easiest way on a Raspberry Pi):

#if you need to install snap 
sudo apt install snapd 
# reboot after snap is installed 
sudo snap install pypy3 --classic

The nice thing about Pypy is that you can use your base Python code as is, and you can do basic testing with Pypy in command line mode.

To run your Python application from the command line substitute python with pypy (or pypy3).

$ pypy3 myapp.py

If you running your Python app as an executable script (i.e. with the chmod +x myapp) then change the first line of your script from #!/usr/bin/python to : #!/usr/bin/pypy3

Pypy Libraries and Limitations

Pypy has an excellent selection of supported libraries , but it is important to note that not all Python libraries are supported. It’s important to check to see if your required library in supported. Unfortunately TKinter is not in the supported list.

To load a Python library into Pypy3, the Python package installer (pip) module needs to be installed:

$ wget https://bootstrap.pypa.io/get-pip.py
$ pypy3 get-pip.py

Once Pypy3 has pip installed, Python packages can be loaded:

$ pypy3 -m pip install some-pymodule

I found that I was able to load some of the “lighter” modules such as: bottle, requests, beautifulsoup4, and pika etc. without any issues. However some of the “heavier” modules such as Numpy and MatPlotLib would not load directly. If you’re planning on using some of the “heavier” Python modules it is recommended that Pypy be run in virtualenv.

Nuitka – a Python compiler written in Python

Nuitka is the Python compiler. It is written in Python. It is a seamless replacement or extension to the Python interpreter and compiles every version of Python (2.6, 2.7, 3.3, 3.4, 3.5, 3.6, 3.7, and 3.8).

Nuitka has two requirements: 1) a “C” compiler, and 2) Python must be installed on the target machine.

Nuitka works on Linux, MacOS X and Window. For Linux the gcc compiler is the default (5.1 or later). The clang compiler is used on macOS X, and for Windows MinGW64 or Visual Studio 2019 compilers can be used.

To install Nuitka:

python -m pip install nuitka

To compile a test project (test1.py) simply enter:

python -m nuitka test1.py

The Nuitka compiled program will be test1.exe in Windows and test1.bin in Linux.

Performance Testing

Performance testing is very subjective and the results can vary greatly based on so many factors. To have some kind of standard I tried using Python’s pystone benchmark testing utility. I used a Raspberry Pi 3 with a number of cases.

The testing showed some interesting results.

  • The jython (Java VMS Python) interpreter was over 2 times slower than native Python
  • iPython (used in Jupyter notebooks) was almost the same speed at native Python
  • PyInstall (a cool Python packager) was 33% slower than native Python
  • A Nuitka executable was 30% faster than native Python
  • Pypy is 9 times faster than native Python and 6.5 times faster than Nuitka

Test Results with pystone.py

TestSpeedFactor
Jython 50000 passes = 5.406222.13
Python3 50000 passes = 2.53779s1
iPython 50000 passes = 2.547941
PyInstall package50000 passes = 3.38577s1.33
Nuitka Executable 50000 passes = 1.78449s0.70
Pypy3 50000 passes = 0.272579s0.11

Summary

There are a number of other choices that can be used to make Python code run faster, (such as Pyston and Cython) but I found Pypy and Nuitka to be the best supported.

Pypy is incredibly fast, and if you’re using “lighter” weight Python modules its a fantastic fit. I think that Pypy would be great for custom microWeb server and SQLite apps. However for data mining and AI projects you might have some issues getting Pypy working with these “heavier” Python modules.

I was super impressed with Nuitka it compiled a variety of my projects without any issues. It had a nice speed improvement. Nuitka also had no problem with GUI libraries like Tkinter, PySimpleGUI and tk_tools or data management libraries like Numpy or Pandas. However Nuitka isn’t bulletproof and I had some problems with video libraries (cw2).

Using AWK in Bash Scripts

In the past Python has been my go to language for quick scripts, however lately I’ve done a lot of projects where I’ve needed to use small Bash scripts.

I found that by adding a little bit of AWK to my Bash scripts I’ve been able to do something in one line of Bash/AWK that would of taken me multiple lines of Python.

AWK is named after it’s authors: Alfred Aho, Peter Weinberger, and Brian Kernighan, and it is an old school (1994) text extraction and reporting tool.

The nice thing about AWK is that you only need to learn a couple of commands to make it usefully.

A Bash/AWK Example

I was setting up a home automation system using a Raspberry Pi with Home Assistant. I was loading a lot of add-on components and I was worried that I might be overloading the Pi so I wanted to monitor the CPU idle time.

Home Assistant supports a command line sensor, so I just needed to get the idle time out of the iostat command:

The idle time value was the 6th item on the 4th line. The AWK code to get only the idle time is:

~$ iostat | awk '{if (NR==4) print $6}'
96.92

AWK logic need to be in single quotes and curly brackets groups together statements. This logic says: if the Number of Record (NR) variable is 4 print the 6th item.

Once I was able to get the idle time I was able to add a sensor in the HA (/config/configuration.yaml) that could be monitored and alarmed:

 sensor:

 platform: command_line 
 name: Idle Time
 command: "iostat | awk 'NR==4' | awk '{print $6}'"
 unit_of_measurement: "%" 

Some Useful AWK Statements

There is an enhanced version of AWK, GAWK (GNU AWK) that might already be loaded on your Linux system. If you are on a Raspberry Pi you can install GAWK by:

sudo apt-get install gawk

There are some excellent tutorials on AWK below are some of commands that I’ve found useful:

substr(string,position,length) – get part of a string:

An example of substr could be used to get the CPU temperature from the sensors utility:

~$ sensors | grep CPU | awk '{print substr($2,2,4)}'
 44.0

The substr() command looks at the 2nd item (+44.0°C), and starts at the 2nd character and it gets 4 characters.

print() with if() – print based on conditions:

The AWK print statement can be used with an if statement to show a filtered list.

An example of this would be to filter the ps (snapshot of the current processes) command, and print only lines with a time showing:

~$ # SHOW ALL PROCESSES
~$ ps -e 
   PID TTY          TIME CMD
     1 ?        00:00:03 systemd
     2 ?        00:00:00 kthreadd
     4 ?        00:00:00 kworker/0:0H
     6 ?        00:00:00 mm_percpu_wq
     7 ?        00:00:00 ksoftirqd/0
     8 ?        00:01:10 rcu_sched
...
~$  # SHOW ONLY PROCESSES WITH TIME
~$ ps -e | awk '{if ($3 != "00:00:00") print $0}'
   PID TTY          TIME CMD
     1 ?        00:00:03 systemd
     8 ?        00:01:10 rcu_sched
    10 ?        00:00:06 migration/0
    15 ?        00:00:03 migration/1
...

systime() / strftime() – get time and format time:

These time functions allow you to add time stamps and then do formatting on the date/time string. I found this useful in logging and charting projects. An example to add a time stamp to the sensor’s CPU temperature would be:

$ sensors | grep CPU | awk '{print strftime("%H:%M:%S ",systime()) $1 $2 }'
 11:06:18 CPU:+45.0°C

Final Comments

I’ve found that learning a little bit of AWK has really paid off.

AWK supports a lot of functionality and it can be used to create full on scripting applications with user inputs, file I/O, math functions and shell commands, but despite all this I’ll stick to Python if things get complex.

STOMP Protocol with RabbitMQ, Node-Red and Python

The STOMP (Simple Text Oriented Messaging Protocol) is a messaging system that is similar to MQTT (Message Queue Telemetric Transport) and AMQP (Advanced Message Queue Protocol).

STOMP uses a send and subscribe mechanism to pass messages and like other messaging systems, the STOMP server is called a broker. Open Source STOMP brokers are available with ActiveMQ and RabbitMQ.

In this blog I wanted to document my setup that used the STOMP plug-in to RabbitMQ and show examples of getting Python and Node-Red subscribing and sending data.

STOMP Specification logo

Why use STOMP?

MQTT and AMQP are very commonly used messaging protocol, so why use STOMP?

AMQP has a lot of great features but for many simple IoT (Internet of Things) applications AMQP may be overkill. Also AMQP is not supported with any mainstream Arduino libraries whereas Arduino Libraries exist for STOMP.

Both STOMP and MQTT are really light-weight, however STOMP has the advantage of being able to add headers and properties along with the message (a feature missing in MQTT but available in AMQP). So you could send a STOMP sensor message and include things like: quality, status, area, and comments along with the value.

STOMP messages can also be used on client side JavaScript web pages. This isn’t supported on AMQP but it possible using Websockets with MQTT.

 

RabbitMQ Setup with STOMP

RabbitMQ is a very popular open source messaging middle-ware. By default RabbitMQ loads AMQP (Advanced Message Queue Protocol) but it can also load MQTT and STOMP protocols. RabbitMQ offers Web Administration and an environment that allows different messaging protocols to intercommunicate.

RabbitMQ can be installed on Window, Linux, MacOS systems and there are also some cloud based offerings. For small systems lower end hardware like a Raspberry Pi can be used.  For complete RabbitMQ installation instructions see: https://www.rabbitmq.com/download.html . To install and run RabbitMQ on a Ubuntu system enter:

sudo apt-get update
sudo apt-get install rabbitmq-server
sudo service rabbitmq-server start

The next step is to add some plug-ins. For this project I loaded the STOMP plug-in and the Web Administration plug-in:

sudo rabbitmq-plugins enable rabbitmq_stomp
sudo rabbitmq-plugins enable rabbitmq_management

The rabbitmqctl command line tool allows you to configure and review the RabbitMQ server. To add a user admin1,  with password admin1, that has config, write and read rights for management and administrator access, enter:

sudo rabbitmqctl add_user admin1 admin1
sudo rabbitmqctl set_permissions -p / admin1 ".*" ".*" ".*"
sudo rabbitmqctl set_user_tags admin1 management administrator

At this point a super-user (admin1) has been defined and that user has been given rights to talk to all features of RabbitMQ including STOMP. See the STOMP documentation for STOMP specific user and system configuration.

After you’ve defined an administrative user the RabbitMQ web management plug-in can be accessed by: http://ip_address:15672 . The Overview tab should show STOMP running on the default port of 61613.

rabbitMQ_ov

For clarity it is good to define a RabbitMQ exchange that is to be used for STOMP. There are a lot of options but the easiest thing is to make a DIRECT exchange that is DURABLE (available after a restart).

rabbitMQ_ex

Python and STOMP

Python has a library component : stompy.py that is both a standalone STOMP testing client and a library component. To install the component:

 pip install stomp.py

Once loaded stomp.py can be used to connect to a server and test subscribing and send data. Below is an example connecting to host 192.168.0.105 with user: pete and password pete.

pete@lubuntu:~$ stomp -H 192.168.0.105 -U pete -W pete


> subscribe /exchange/stomp1/tag1
Subscribing to '/exchange/stomp1/tag1' with acknowledge set to 'auto', id set to '1'
> send /exchange/stomp1/tag1 hi from python
> 
message-id: T_1@@session-8xhCjj0giLAiOFg8E6OGPQ@@1
subscription: 1

hi from python
> 

For this example the exchange stomp1 was defined. The topic tag1 was dynamically defined and a message was sent and subscribe to. The topic (tag1) doesn’t need to be defined in RabbitMQ.

A Python STOMP test program would be:

 #  
 # stomp_test.py - test STOMP  
 #  
 import time  
 import sys  
   
 import stomp  
 import random  
   
 class MyListener(stomp.ConnectionListener):  
   def on_error(self, headers, message):  
     print('received an error "%s"' % message)  
   def on_message(self, headers, message):  
     print('received a message "%s"' % message)  
   
 # Define a STOMP connection and port  
 conn = stomp.Connection([("192.168.0.105", 61613)])  
 conn.set_listener('', MyListener())  
 conn.connect('pete', 'pete', wait=True) # define the username/password  
   
 # Setup a subscription  
 conn.subscribe(destination='/exchange/stomp1/tag1', id=1, ack='auto')  
   
 while True:  
   time.sleep(15) # send a random message every 15 seconds  
   conn.send(body=str(random.randint(1,11)), destination='/exchange/stomp1/tag1')  
   
 #conn.disconnect()  

 

Node-Red and STOMP

The Node-Red STOMP node can be installed using the menu option “Manage Palette”  or manually:

cd $HOME
cd ~/.node-red 
npm install node-red-node-stomp

A simple Node-Red test circuit would include:

  • an injector node – manually send in a test string
  • a STOMP output node – to send messages to a broker
  • a STOMP input node – subscribe to messages
  • a debug node – to show messages in the debug tab

nr_sample

The logic requires the STOMP node to be defined, and the STOMP nodes need to have a valid exchange and topic:

Node-Red can be used to configure Web dashboards that can show trending and charting of STOMP values.

 

Testing STOMP with RabbitMQ

Once I had both Python and Node-Red working with STOMP I was able to go back to my RabbitMQ and view the connections.

rabbitMQ_conn

It is also possible to use RabbitMQ to publish STOMP messages. In this example I included in the header: status, quality and units.

rabbitMQ_ex_pub

I then was able to check the message in Node-Red.

nr_msg_debug

 

Final Comments

Compared to AMQP and MQTT there aren’t nearly as many examples so I needed to do a little bit of experimentation. As an example I found that if I send and subscribe to a queue (use: /queue/my_queue)  this was only a 1-to-1 connection, if I had 2 subscribers each subscriber would only get 1/2 the messages. So I’m recommend using RabbitMQ exchanges for STOMP messaging.

This was a quick introduction to STOMP with Node-Red and Python. I still need to look at Arduino and Javascript connections.

 

 

 

Julia programming on a Raspberry Pi

Julia is a free and open-source general purpose programming language made specifically for scientific computing.

In this blog I wanted to document my Pi Julia interface testing.

Getting Started

Julia is supported on Windows, MacOS and Linux. See the download documentation for your specific system at : https://julialang.org/downloads/. To install Julia on a Raspberry Pi enter:

sudo apt install julia

To check your install, you can run Julia at the command line:

pi@raspberrypi:~ $ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.0.3
 _/ |\__'_|_|_|\__'_|  |  Raspbian ⛬  julia/1.0.3+dfsg-4+rpi1
|__/                   |

julia> 

Julia packages can be installed at the command line by:

julia> using Pkg
julia> Pkg.add("some package")

Julia with Raspberry Pi GPIO

A Raspberry Pi package “PiGPIO” is installed by:

julia> using Pkg
julia> Pkg.add("PiGPIO")

Once the package is installed a daemon ( pigpiod) is created that the Julia script connects to. The pigpiod daemon is started by:

sudo pigpiod

Below is an example script that cycles a GPIO pin. For this example the GPIO pin is passed as a keyboard input. The readline() method returns a string, so parse() is used get an integer.

# pi1.jl - GPIO test 
using PiGPIO 

print("Enter the GPIO pin: ")

n = readline()
led_pin = parse(Int64,n) # convert the string input to an Integer

println("Pin used: $led_pin ")


#led_pin = 4 #use BCM pin 4 (physical pin 7) 
p=Pi() #connect to pigpiod daemon on localhost 
set_mode(p, led_pin, PiGPIO.OUTPUT) 
println("Julia is cycling an LED 5 times") 

try 
    for i in 1:5 # cycle 5 times 
        PiGPIO.write(p, led_pin, PiGPIO.HIGH) 
        sleep(0.5) 
        PiGPIO.write(p, led_pin, PiGPIO.LOW) 
        sleep(0.5) 
    end 
finally 
    println("Cleaning up!") 
    # if you wish to leave the pin as an input 
    #set_mode(p, led_pin, PiGPIO.INPUT) 
end

To run the code enter:

pi@raspberrypi:~ $ julia pi1.jl
Enter the GPIO pin: 4
Pin used: 4
[ Info: Successfully connected!
Julia is cycling an LED 5 times
Cleaning up!

Raspberry Pi Inputs

A simple GPIO read input programs is:

# pi2.jl - GPIO input test 

using PiGPIO

p = Pi() #connect to pigpiod daemon on localhost 

in_pin = 23
set_mode(p, in_pin, PiGPIO.INPUT)

println("\nInput status on Pin 23...")
println("hit Cntl-C to break out")

println(PiGPIO.read(p, in_pin))

while true
    println(PiGPIO.read(p, in_pin))
    sleep(1)
end

 

The output will look something like:

pi@raspberrypi: $ julia pi2.jl
[ Info: Successfully connected!

Input status on Pin 23...
hit Cntl-C to break out
1
1
0
0

Final Comments

As a Python user l found using Julia to be  frustrating, the documentation is weak and there aren’t the extensive libraries that I’m used to using in Python.

Neopixel Hats

There are some fun wearable projects that you can do with small Arduino modules like the Gemma ($9) and some neopixels. ($7).

 

Conductive thread is used to both secure the modules to the material and to do the wiring.

For this project we wanted to have a sequence of colour patterns, so we had:

  • a rainbow moving around in a circle
  • a red smiley face, that had a left/right smirk and then a wink
  • a rainbow moving around in a circle
  • a red smiley face, that had a left/right smirk and then a wink

However there a tons of possible patterns that could be used.

For the battery mounting there are a few options such as lipo batteries, coin batteries and small battery packs.

Below is some example code for the moving rainbow that we used.

Have Fun

#include <Adafruit_NeoPixel.h>

#define PIN 1
int theLED = 0;


Adafruit_NeoPixel strip = Adafruit_NeoPixel(12, PIN, NEO_GRB + NEO_KHZ800);

void setup() {
  strip.begin();
  strip.setBrightness(20);
  strip.show(); // Initialize all pixels to 'off'
}

void loop() {
  // Create a rainbow of colors that moves in a circle 

  int col;
  
     for (int i=0; i < strip.numPixels(); i++) {
        strip.setPixelColor(i, 0, 0, 255);
     }
    theLED = theLED + 1;
    if (theLED >= strip.numPixels()) {
      theLED = 0; 
    }
    strip.setPixelColor(theLED, 209, 31, 141);
    strip.setPixelColor(theLED - 1, 11, 214, 180);
    strip.setPixelColor(theLED - 2, 240, 210, 127);
    strip.setPixelColor(theLED - 3, 191, 127, 240);
    strip.show();
    delay (500);
}