Same Code for Python GUI and Web Apps

The Python PySimpleGUI project has two main goals:

  • A simpler method for creating graphical interfaces, and
  • Common code for Tkinter, QT, xW and Web graphics

I feel comfortable doing my own Tkinter and Web interfaces, but common code for both is very cool and it could be very useful for Rasp Pi applications. However things didn’t work “out of the box”, so this blog will look at some workarounds and issues.

A Simple Example

If you are used to coding in QT or xW PySimple is absolutely easier to use.

PySimpleGUI has some defined graphic objects (like Text or Button) and the screen placement is defined by how they are grouped with [ ] brackets. For example two objects in a [ ] bracket puts the two objects on the same line. 

If you’re a Tkinter user PySimpleGUI may seem a little more cumbersome in it’s layout design. (I find Tkinter’s grid methods to be a superior layout approach).

For my first example I used the published example and I added an option for the graphic method (tkinter, Qt, Wx and Web):

 import sys, random, time  
   
 mode = "tkinter"  
 # Check for command line arguments   
 if len(sys.argv) > 1: # if there is use the Web Interface  
   if sys.argv[1] == 'web':  
     import PySimpleGUIWeb as sg  
     mode = "Web"  
   elif sys.argv[1] == 'wx':  
     import PySimpleGUIWx as sg  
     mode = "Wx"  
   elif sys.argv[1] == 'qt':  
     import PySimpleGUIQt as sg  
     mode = "Qt"  
   else:  
     import PySimpleGUI as sg  
   
 else: # if no arguments use the standard GUI  
   import PySimpleGUI as sg  
   
 # Basic example from PySimpleGUI
   
 sg.change_look_and_feel('DarkAmber')  # Add a touch of color  
 # All the stuff inside your window.  
 layout = [ [sg.Text('Some text on Row 1')],  
       [sg.Text('Enter something on Row 2'), sg.InputText()],  
       [sg.Button('Ok'), sg.Button('Cancel')] ]  
   
 # Create the Window  
 window = sg.Window('PySimpleGUI ' + mode , layout)  
 # Event Loop to process "events" and get the "values" of the inputs  
 while True:  
   event, values = window.read()  
   if event in (None, 'Cancel'):  # if user closes window or clicks cancel  
     break  
   print('You entered ', values[0])  
   
 window.close()  
 print('Program terminating normally')  
   

The output for the different graphic methods looks incredibly similar for this example:

GUI_wxGUI_webGUI_tkGUI_qt

It’s important to note that PySimpleGUI is still under development so many of the features will be in transition. If you stick to basic functions like: buttons, images and text you should be good.

I found that the default Tkinter version to be very solid, and the other versions would complain on some of the added features such as: frames, sliders, and progressbars.

A Realtime Update Example

For my next example I wanted to do a Rasp Pi type example with some simulated values from a temperature sensor.

To write back to a graphic object:

  • define a key name when the object is defined (key = “myojbname”)
  • use:  window[‘myojbname’]( new_value) to update the object
 import sys, random, time  
   
 mode = "tkinter"  
 # Check for command line arguments   
 if len(sys.argv) > 1: # if there is use the Web Interface  
   if sys.argv[1] == 'web':  
     import PySimpleGUIWeb as sg  
     mode = "Web"  
   elif sys.argv[1] == 'wx':  
     import PySimpleGUIWx as sg  
     mode = "Wx"  
   elif sys.argv[1] == 'qt':  
     import PySimpleGUIQt as sg  
     mode = "Qt"  
   else:  
     import PySimpleGUI as sg  
   
 else: # if no arguments use the standard GUI  
   import PySimpleGUI as sg  
   
 # Basic example of PSGWeb  
   
 def main():  
   sg.change_look_and_feel('LightGrey')  
   dataframe = [  
     [sg.Text('Temperature (C):',size=(12,2)),  
     sg.Text(' ', font='Courier 48', text_color='red', key='degC')],  
     [sg.Text('Humidity (%)  :',size=(12,2)),  
     sg.Text(' ', font='Courier 48', text_color='red', key='humid')],  
       
   ]  
   layout = [  
     [sg.Image(filename = "dht11.png"),sg.Frame("DHT11 Sensor", dataframe,font='Any 12', title_color='blue')]  
     ]  
     
   if mode == "web":  # Note: customization on the Web interface
     window = sg.Window('Pi Weather Data ' + mode, layout,web_ip='192.168.0.107', web_port = 8888, web_start_browser=False, disable_close=True)  
   else:  
     window = sg.Window('Pi Weather Data ' + mode, layout)  
     
   while True:  
     event, values = window.read(timeout=10)  
     temp_degC = random.randint(20,30)  
     humidity = random.randint(0,100)  
     window['degC']( temp_degC)  
     window['humid'](humidity)  
     time.sleep(1)  
       
   window.close()  
   
 main()  
 print('Program terminating normally')  
   

The Wx code would not run, and the development team has commented on compatibility issues that they are having with Wx so future support may be dropped.

The Tkinter and Qt, but for this example the Web presentation dropped the frame outline.

GUI2_qtGUI2_tkGUI2_web

PySimpleGUIWeb

The PySimpleGUIWeb project is based on the Remi Project which defines graphics objects in an HTML/Web environment.

One big difference that I found between the Remi project and PySimpleGUIWeb is that Remi supports callbacks. I found that callbacks are extremely useful if you are trying to mix reads and writes in the same application. PySimpleGUI uses a read to get user input but this gets confusing if you are using a timer for writing new data to the GUI.

The PySimpleGUIWeb library by default was missing some key documentation. I found that “out of the box” it wasn’t usable for my applications . However everything that I needed was in the Window class it just wasn’t clearly documented.

The following web features can be controlled:

  • web_ip=’xx.xx.xx.xx’ , default is localhost (127.0.0.1), not good for remote connections.
  • web_port= xxxx , default is a random port id. Again not useful for remote connections.
  • web_start_browser=False, default a web browser is launched (not useful for SSH).
  • disable_close=True , default closes the app when the web browser closes.

An example to open a Web connection (not local) that doesn’t auto-open a browser and stays open would be:

window = sg.Window('Pi Weather Data ',  layout,web_ip='192.168.0.107', web_port = 8888, web_start_browser=False, disable_close=True)

Final Thoughts

I like the concept of the same code being used on both a Tkinter and a Web  app. I believe that this has a lot of potential. However at this point only the simplest features are available in the Web, so I’d rather use a Python Web framework like Bottle.

I really like some of the concepts of PySimpleGUI but I think that understanding and using the standard Tkinter allows you more flexibility and it lets you include add-in modules like Tk_tools.

 

Pi Appliance

My goal was to make a Pi kitchen appliance that shows me the key things that I want to see and music I want to listen to while I’m getting my morning coffee. For this project I used a Rasp Pi with 2.8″ TFT touchscreen. These screens start at a round $15.

People’s morning interests will vary so in this blog I just wanted to highlight some of the issues that I needed to worked through. For me the main stumbling blocks were:

  • Hiding the top Rasp Pi menu bar
  • Creating a GUI that uses the full screen
  • Getting weather data
  • scraping web pages to extract what I need

Getting Started

There are some great Raspberry Pi TFT screens that come with buttons and cases. You will need to look at the documentation that comes with your screen, but a good reference is: https://learn.adafruit.com/adafruit-pitft-28-inch-resistive-touchscreen-display-raspberry-pitft_case

For my project I simply used some of my kids Lego blocks.

pi_kitch2

Remove the Pi Menu Bar

The Pi TFT screen isn’t super large, so I wanted to remove the Pi menu bar and run my application at full size.

tft_w_menu

To remove the menu bar tweek two files. First:

sudo nano /etc/xdg/lxsession/LXDE-pi/autostart

Comment out the line  (with #) :

@lxpanel --profile LXDE

Then do the same for:

nano /home/pi/.config/lxsession/LXDE-pi/autostart

After this reboot the system.

Create a Full Size App

There are a multitude of choices for a screen layout. I was looking for lines of text, with maybe the bottom line used for buttons. I found that 7 lines was a reasonable fit. To remove the Python Tkinter title I positioned the top of the screen above the physical screen position (-30 instead of 0).


# My Kitchen Appliance App
#
import urllib.request as urllib2
import tkinter as Tkinter
from tkinter.ttk import *

from tkinter.font import Font
from tkinter import messagebox
top = Tkinter.Tk()
top.title("My Kitchen Appliance")
top.geometry("320x240+-5+-30") # set screen size, left (-5) and top (-30)
top.resizable(False, False)
top.details_expanded = False

#Define the buttons
myfont = Font(family="Times New Roman Bold",size= 12) # Should try a few more sizes

tft_rows = 7 # try 7 rows of buttons
tftbutton = ['' for i in range(tft_rows)]
for i in range(tft_rows):
    tftbutton[i] = Tkinter.Button(top, text = "Line " + str(i+1), fg = "blue", bg = "white", anchor="w", width= 35, height= 1,font=myfont).grid(row=(i+1),column=1) # a buttpn arra

top.mainloop()

The Python GUI will look like this:

tft_7bttns

Get Weather Data

There are a number of good weather API’s. I used OpenWeather because I can use it in variety of apps like Node-Red. OpenWeather has a free user API but you should login and get an appid.

A Python example to get some current weather data for a city:


# get Open Weather (REST API) data
import requests

# api-endpoint

URL = "https://openweathermap.org/data/2.5/weather?q="
mycity = "burlington,CA"
myappid = "&appid=b6907d289e10d714a6e88b30761fae22"
# sending get request and saving the response as response object
fullURL = URL + mycity + myappid
r = requests.get(fullURL)

# extracting data in json format
data = r.json()

print (data)

# Check out the structure
#for index, value in enumerate(data):
# print(index, value)

# Show some weather data
print (data['weather'][0]['description'])
print (data['weather'][0]['main'])
print (str(int(data['main']['temp'])) + " C")
# convert wind speed from meters/sec to kph
print (str((data['wind']['speed'] * 3.6)) + " kph")

This code will give output such as:

$Python3 burlweather.py
{'coord': {'lon': -79.8, 'lat': 43.32}, 'weather': [{'id': 803, 'main': 
'Clouds', 'description': 'broken clouds', 'icon': '04n'}], 'base': 
'stations', 'main': {'temp': 5.81, 'pressure': 1014, 'humidity': 93, 
'temp_min': 3.33, 'temp_max': 7.78}, 'visibility': 24140, 'wind': 
{'speed': 2.1, 'deg': 50}, 'clouds': {'all': 75}, 'dt': 1574816701,
 'sys': {'type': 1, 'id': 818, 'country': 'CA', 'sunrise': 1574771158, 
'sunset': 1574804839}, 'timezone': -18000, 'id': 5911592, 'name': 'Burlington', 'cod': 200}
broken clouds
Clouds
5 C
7 kph

Scraping Web Pages

I wasn’t able to find an API for all the things I was after, so I need to scrape web pages. The Python Beautiful Soup library is a great for finding and grabbing stuff on web pages. To install it:

$ apt-get install python-bs4 (for Python 2)

$ apt-get install python3-bs4 (for Python 3)

I had an example where I wanted to find the ski lifts and runs open. I had the Web page but I needed to search the ugly HTML code.

ski_bs0

ski_bs

In the HTML code I found that the lift and run information is contained in a <p class=“open value” tag. Beautiful Soup allows you to make searches based on attributes. The results can be HTML code or the .text property will return the results as simple text (no HTML code).

The following Python code would search my URL and extract the number of lifts open:


$ python
Python 3.7.4
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.request as urllib2
>>> from bs4 import BeautifulSoup
>>> theurl = 'https://www.onthesnow.ca/ontario/blue-mountain/skireport.html'
>>> page = urllib2.urlopen(theurl)
>>> soup = BeautifulSoup(page, 'html.parser')
>>> liftopen = soup.find("p", attrs={"class":"open value"})
>>> liftopen.text
'2 of 11'

Final Comments

There are a ton of different “Pi Appliance” applications that could be done. I hope that some of these hints that I’ve documented are helpful.

pi_kitch1

Pan-Tilt-Shoot Webcam

PTZ or Pan-tilt-zoom cameras are off the shelf and reasonably low cost. This project is a home made pan-tilt camera that shoots nerf rockets.

I got the rocket launcher in a bargain bin, but they can be found online starting around $15.

For this project there were 3 main steps:

  • getting the Python code talking to the rocket launcher
  • loading some USB webcam software
  • loading a Python web framework (bottle).

The hardware could be a PC (preferably running Linux) or a Raspberry Pi. On this project I used a Pi clone (an Orange Pi Lite). The faster your hardware the better the streaming video performance that you’ll get.

pts_overview

Getting the Rocket Launcher Working

There are a number of Python libraries that need to loaded:

pip install setuptools
pip install usb

To control the rocket launcher Python can connect to the USB device using the vendor id (0x2123) and product id (0x1010). Rocket launcher commands are issued as USB transfer codes.

A command line test program (rocket1.py) would be:

 import usb  
 import sys  
 import time  
 device = usb.core.find(idVendor=0x2123, idProduct=0x1010)  
 # On Linux we need to detach usb HID first  
 try:  
   device.detach_kernel_driver(0)  
 # except Exception, e:  
 except Exception:  
   pass # already unregistered  
 device.set_configuration()  
 endpoint = device[0][(0,0)][0]  
 down = 1 # down  
 up = 2 # up  
 left = 4 # rotate left  
 right = 8 # rotate right  
 fire = 16 # fire  
 stop = 32 # stop  
 #device.ctrl_transfer(0x21, 0x09, 0x0200, 0, [signal])  
 while True:  
   print('r = right, l = left, u = up, d = down, f = fire ')  
   key = raw_input ('enter key:')  
   if (key == 'l'):  
     device.ctrl_transfer(0x21, 0x09, 0, 0, [0x02, left, 0x00,0x00,0x00,0x00,0x00,0x00])  
   if (key == 'u'):  
     device.ctrl_transfer(0x21, 0x09, 0, 0, [0x02, up, 0x00,0x00,0x00,0x00,0x00,0x00])  
   if (key == 'r'):  
     device.ctrl_transfer(0x21, 0x09, 0, 0, [0x02, right, 0x00,0x00,0x00,0x00,0x00,0x00])  
   if (key == 'd'):  
     device.ctrl_transfer(0x21, 0x09, 0, 0, [0x02, down, 0x00,0x00,0x00,0x00,0x00,0x00])  
   if (key == 'f'):  
     device.ctrl_transfer(0x21, 0x09, 0, 0, [0x02, fire, 0x00,0x00,0x00,0x00,0x00,0x00])  
     time.sleep(4)  
   time.sleep(0.1)  
   device.ctrl_transfer(0x21, 0x09, 0, 0, [0x02, stop, 0x00,0x00,0x00,0x00,0x00,0x00])  

By default the USB port requires superuser rights, so run the program using sudo:

$ sudo python rocket1.py
r = right, l = left, u = up, d = down, f = fire
enter key:r
r = right, l = left, u = up, d = down, f = fire
enter key:

Getting the Web Cam working

A standard USB Web Cam can be connected to a Linux  PC or Raspberry Pi using the  motion package. Motion is super easy to setup and it’s got lots of added features if you want to look at enhancing things later. To install motion:

sudo apt-get install motion

Once you have motion installed you’ll need to tweek some of it’s parameters by:

sudo nano /etc/motion/motion.conf

The /etc/motion/motion.conf file contains a lot of  parameters, some of the more important ones are:

# Image width (pixels). Valid range: Camera dependent, default: 352
width 800

# Image height (pixels). Valid range: Camera dependent, default: 288
height 600

# Maximum number of frames to be captured per second.
framerate 1

# Maximum framerate for stream streams (default: 1)
stream_maxrate 1

# Restrict stream connections to localhost only (default: on)
stream_localhost off

The speed of your hardware and network will determine how many frames per second you can use.

To run the video server enter:

sudo motion &

The motion package has a built in web server that is accessed by: http://your_ip:8081

livevideo

Getting the Bottle Web Server Running

There are a lot of web programming options that are available. In my case I wanted to run the project as a simple standalone app on an Orange Pi (a Raspberry Pi clone). To install bottle:

sudo apt-get install python-bottle

The Pan-Tilt-Shoot Web page (ptscam.html) was designed with 5 buttons and the Web Cam video below. When a button is clicked a javascript function called sendcmd passes a command as a header item in a AJAX request.  The motion web server camweb video is included using the html img tag, with the source being the web link.

<!DOCTYPE html>
<html>
<head> 
<title>Pan-Tilt-Shoot Webcam</title> 
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="icon" href="data:,"> 
<style>
  html{font-family: Helvetica; display:inline-block; margin: 0px auto; text-align: center;}
  h1{color: #0F3376; padding: 2vh;}p{font-size: 1.5rem;}
  .button{display: inline-block; background-color: #4286f4; border: none; 
  border-radius: 4px; color: white; text-decoration: none; font-size: 30px; width:100%}
  .button2{background-color: green ;width:31%}
  .button3{background-color: red; width:33%}
</style>
</head>
<script>
function sendcmd(thecmd) {
  // send the action as a header item 
  var xhttp = new XMLHttpRequest();
  xhttp.open("GET","/action" , true);
  xhttp.setRequestHeader("myaction", thecmd);
  xhttp.send()
}
</script> 
<body>
<h2>Pan-Tilt-Shoot Webcam</h2> 
<button onclick="sendcmd('up')" class="button">UP</button>
<button onclick="sendcmd('left')" class="button button2">LEFT</button>
<button onclick="sendcmd('fire')" class="button button3">FIRE</button>
<button onclick="sendcmd('right')" class="button button2" >RIGHT</button>
<button onclick="sendcmd('down')" class="button button">DOWN</button>
  
<p><img src='http://192.168.0.117:8081/'></p>
</body>
</html>

The Python web server application sends the web page at startup, and then it processes AJAX requests and passed the requested action (as a header item) to the rocket launcher USB device.

 # Python Bottle   
 #  
 import os, socket  
 from bottle import route, run, static_file, request  
 import usb  
 import sys  
 import time  
    
   
 # Send an action to the rocket launcher  
 def do_action(theaction):  
    
   print("Action : " + theaction)  
   down = 1 # down  
   up = 2 # up  
   left = 4 # rotate left  
   right = 8 # rotate right  
   fire = 16 # fire  
   stop = 32 # stop  
   if (theaction == 'left'):  
     device.ctrl_transfer(0x21, 0x09, 0, 0, [0x02, left, 0x00,0x00,0x00,0x00,0x00,0x00])  
   if (theaction == 'up'):  
     device.ctrl_transfer(0x21, 0x09, 0, 0, [0x02, up, 0x00,0x00,0x00,0x00,0x00,0x00])  
   if (theaction == 'right'):  
     device.ctrl_transfer(0x21, 0x09, 0, 0, [0x02, right, 0x00,0x00,0x00,0x00,0x00,0x00])  
   if (theaction == 'down'):  
     device.ctrl_transfer(0x21, 0x09, 0, 0, [0x02, down, 0x00,0x00,0x00,0x00,0x00,0x00])  
   if (theaction == 'fire'):  
     device.ctrl_transfer(0x21, 0x09, 0, 0, [0x02, fire, 0x00,0x00,0x00,0x00,0x00,0x00])  
     time.sleep(4)  
   time.sleep(0.1)  
   device.ctrl_transfer(0x21, 0x09, 0, 0, [0x02, stop, 0x00,0x00,0x00,0x00,0x00,0x00])  

# Send the starting page   
 @route('/')  
 def server_static():  
   return static_file("ptscam.html", root='')  
# Process an AJAX GET request, pass the action to the rocket launcher code   
 @route('/action')  
 def get_action():  
   print("Requested action: " + request.headers.get('myaction'))  
   do_action(request.headers.get('myaction'))  
       
   
 # On Linux we need to detach usb HID first  
 device = usb.core.find(idVendor=0x2123, idProduct=0x1010)  
 try:  
   device.detach_kernel_driver(0)  
 except Exception:  
   pass # already unregistered  
   
 # Start the bottle web server  
 run(host="192.168.0.117" , port=8000, debug=False)  

Because of the USB connection the Python application need to be run under sudo. When the program is running some diagnostics will show connections and actions.

$ sudo python ptscam.py
Bottle v0.12.13 server starting up (using WSGIRefServer())...
Listening on http://192.168.0.117:8000/
Hit Ctrl-C to quit.

192.168.0.114 - - [23/Nov/2019 19:26:24] "GET / HTTP/1.1" 200 1251
192.168.0.114 - - [23/Nov/2019 19:26:24] "GET /:8081 HTTP/1.1" 404 736
192.168.0.114 - - [23/Nov/2019 19:26:53] "GET / HTTP/1.1" 200 1271
Requested action: up
Action : up

The web page is accessed by : http://your_ip:8000

pts_screen

Final Comments

The next step will be to mount the rocket launcher with the USB web cam on a little rover.

Arduino Yún for IoT

There are some great Internet of Things hardware platforms, such as Raspberry Pi’s, ESP-32 based Arduino modules, Particle and Beaglebone to name just a few. Each of these systems have their strengths and weakness. Some are strong on the hardware side, like the Arduino modules, and others excel on the programming side like the Raspberry Pi.  

The Yún is somewhat unique in that it’s a module with two processors, one that supports standard Arduino programming and a second processor that supports Linux and the OpenWrt wireless stack. The Yún ($59) has an Arduino Uno form factor and there are clones like the LinkIt Smart 7688 Duo ($18) in an Arduino Nano form factor.

yun_overview

In this blog I wanted to document some of key feature and functions that I worked through, namely:

  • moving file – scp and ftp
  • Python bridging to Arduino
  • uhttpd Web Server – with Python CGI
  • Yún REST API
  • MQTT
  • Yún mailbox

Yún Overview

The Yún, now in revision 2, seems to have been somewhat overlooked because of the all the low cost ESP-8266 and ESP32 based Arduino modules starting around $2.

Some the interesting features of the Yún include:

  • Arduino code can launch and get feedback from Linux apps
  • A bridge library allows Linux programs and Arduino to share data
  • Arduino libraries for:
    • Web client
    • Web server
    • Mailbox to Linux
  • Yún has a lightweight web server, uhttpd, that can be used to Python, PHP, Lua CGI web programs
  • Yún has a read/write REST API web service

Getting Started

The Arduino side of the Yún is available “out of the box” like any other Arduino module when you connect a USB cable into the module.

The Linux side however requires some configuration. Please see one of the guides on the Linux setup. Once your module is connected to your network, the Yun webserver can be used to configure features and add software components.

yun_software1

Another option for loading software is to use the opkg package manager from an SSH connection.

yun_opkg

So for example to install nano, the simple text editor, enter:

 opkg install nano

Moving File and Working in OpenWrt Linux

The OpenWrt Linux isn’t loaded with with an X-windows environment and this means that you can not run idle or leafpad etc. to do easy editing of program files. Nano is a good command line text editor but it’s not the same as a windows based editor.

To move files between a PC and OpenWrt you have some options. The two that I like are:

  • scp – secure copy, this is built in both OpenWrt and Microsoft Window 10
  • ftp – file transfer protocol. Needs to be installed in OpenWrt.

There are a few ftp servers that could be installed in OpenWrt, vsftp is a lightweight option and it can be installed by:

opkg update
opkg install vsftpd

Once vsftpd is installed it needs to be enabled, this can be done from the command line or via the web interface.

yun_ftp_start.png

Yún Bridge

The Yun Arduino bridge library allow variables to be passed between the Arduino code and a Python program. Below is an example that writes two random variables available, and it creates a “bridge1” variable that can be written from the Python side.

// Simple Yun Bridge Example
//

#include <Bridge.h>
// create a bridge variable to get remote data
char bridge_Value[10];

void setup()
{
    Bridge.begin();     // this launches /usr/bin/run-bride on Linino
}

void loop()
{    
    // create 2 random bridge values, that are sourced from the Arduino
    Bridge.put("random1", String(random(1, 100)));
    Bridge.put("random2", String(random(1, 100)));
    // Called the bridge value "bridge1". This is the name used on the Python side
    Bridge.get("bridge1", bridge_Value, 6);
    delay(1000);  
}

An example to read/write in Python:


#!/usr/bin/python

import sys

sys.path.insert(0, '/usr/lib/python2.7/bridge/')

from bridgeclient import BridgeClient as bridgeclient
value = bridgeclient()

message = value.get("random1") # get a value from Arduino

print "Random1: ", message

value.put("bridge1","1111")  # set a value to Arduino

Yún uhttpd web server

The uhttp web server is used for Yun setup and configuration. This web server can also be used to for custom static pages and user web apps. To view/modify the web server settings:

nano /etc/config/uhttpd

Within this config file, you can enable custom web applications by defining an interpreter:

# List of extension->interpreter mappings.
# Files with an associated interpreter can
# be called outside of the CGI prefix and do
# not need to be executable.
# list interpreter ".php=/usr/bin/php-cgi"
# list interpreter ".cgi=/usr/bin/perl"
list interpreter ".py=/usr/bin/python

# Lua url prefix and handler script.
# Lua support is disabled if no prefix given.
# option lua_prefix /luci
# option lua_handler /usr/lib/lua/luci/sgi/uhttpd.lua

The default directory for user programs is: /www/cgi-bin

Python CGI – Get Values

To read the Arduino bridge values in a Python CGI program, add a file to the /www/cgi-bin directory. For my example I called the file p1.py :


#!/usr/bin/python

import sys
import cgi

sys.path.insert(0, '/usr/lib/python2.7/bridge/')

from bridgeclient import BridgeClient as bridgeclient
value = bridgeclient()

print "Content-type:text/html\r\n\r\n"
print '<html>'
print '<head>'
print '<title>Python CGI Bridge</title>'
print '</head>'
print '<body>'
print '<h2>Python CGI Bridge</h2>'
print 'Random Value1: ' + value.get("random1")
print 'Random Value2: ' + value.get("random2")
print '</body>'
print '</html>'

To run the python script on the web page you will need to change the file right to executable:

chmod +x p1.py

You can debug and see output from the command line.


root@yun1:/www/cgi-bin# ./p1.py
Content-type:text/html

<html>
<head>
<title>Python CGI Bridge</title>
</head>
<body>
<h2>Python CGI Bridge</h2>
Random Value1: 13
Random Value2: 24
</body>
</html>

If the output looks good, try the app from the Web page:

CGI_bridge1

Python CGI – Put Values

There are a number of methods that can be used to send user input from a web page. A simple approach is to use a form. The form data can be read from the cgi.FieldStorage object, using a form.getvalue() call.

#!/usr/bin/python

# Import modules for CGI handling
import cgi
import sys

sys.path.insert(0, '/usr/lib/python2.7/bridge/')
from bridgeclient import BridgeClient as bridgeclient
value = bridgeclient()

# Create instance of FieldStorage
form = cgi.FieldStorage()

# Get data from fields
bridge1 = form.getvalue('bridge1')
if bridge1 != "" :
    value.put("bridge1",bridge1)


print "Content-type:text/html\r\n\r\n"
print """
<html>
<head>
<title>Python CGI - Put Bridge Value</title>
</head>
<body>
<h2>Python CGI Form - Put Bridge Value</h2>
<form action = "/cgi-bin/p2.py" method = "post">
Enter BRIDGE1 value: <input type = "text" name = "bridge1"><br />

<input type = "submit" value = "Submit" />
</form>
</body>
</html>"""

 

The web page will call itself when the submit button is pressed.

CGI_bridge2

Yún REST API

The Yún REST API is a web service that allow remote users and web browsers to view and set bridge values. I found the REST API to be a good tool for testing my Web CGI and Python applications

To view all bridge variables enter: http://yourYun_IP/data/get

yun_rest_1

To get a specific bridge value enter:  http://yourYun_IP/data/get/my_bridge_value

yun_rest_2

To put a bridge value enter: http://yourYun_IP/data/put/my_bridge_value/value 

yun_rest_3

IoT Connections – MQTT

For Internet of Things (IoT) projects you need to pass data from the Arduino and some server. The communications and the server would be something like MQTT or Redis.

The Yún does not have direct access to the Wifi or ethernet port so the standard Arduino libraries for MQTT or Redis etc. will not work. An alternative approach is load the protocol’s command line client on the Linux side and then Arduino can shell out to the Linux tool.

For example to load the  Mosquitto MQTT command line tools:

opkg update
opkg install mosquitto-client

To test MQTT publishing to a topic (mytag1) with a message of 249 on remote client (192.168.0.116) :

mosquitto_pub -h 192.168.0.116 -u username -P password -t mytag1 -m 249

To remotely subscribe to

mosquitto_sub -h 192.168.0.116 -u username -P password -t mytag1

An example of sending out four MQTT messages in Arduino:

/*
Shell out to pass values to a MQTT command line pub
*/
#include <Process.h>

void setup() {
  Bridge.begin();	// Initialize the Bridge
  Serial.begin(9600);	// Initialize the Serial

  // Wait until a Serial Monitor is connected.
  while (!SerialUSB);
}

void loop() {
  Process p;
  String thecmd;
  String strval = String( random(0,100));
  // create a string with host, username and password, -t is for the topic
  String theparams = "-h 192.168.0.116 -u pete -P pete -t ";
  int numtopics = 4;
  String topics[4] = {"tag1","tag2","tag3","tag4"};

  for (int i=0; i < numtopics ; i++) {
    strval = String( (i*100) + random(1,99)); // create a random value - 0,100+,200+,300+
    thecmd = "mosquitto_pub " + theparams + topics[i] + " -m " + strval;
    Serial.println(thecmd);
    p.runShellCommand(thecmd);
    // do nothing until the process finishes, so you get the whole output:
    while (p.running());
  }
  // Wait 5 seconds and repeat
  delay(5000);  // wait 5 seconds before you do it again
}

It is also possible to create some bridge variables and pass them to a Python program that could do the MQTT communications.

Yún Mailbox

At present the mailbox only works in one direction, and this into the Arduino. Both the REST web interface and Python have writing functionality but no read capabilities (on the forums this has been identified, so a future revision may add this).

To write using REST:

http://yun_ip/mailbox/mymessagetext

To write using Python:


import sys
sys.path.insert(0, '/usr/lib/python2.7/bridge')
from bridgeclient import BridgeClient
client = BridgeClient()
client.mailbox("my_message")

The Arduino code to read message:

// Mailbox Read Example
//
#include <Mailbox.h>

void setup() {

  // Initialize Bridge and Mailbox
  Bridge.begin();
  Mailbox.begin();

  // Initialize Serial
  SerialUSB.begin(9600);

  // Wait until a Serial Monitor is connected.
  while (!SerialUSB);
  SerialUSB.println("Mailbox Read Message\n");
}

void loop() {
  String message;

  // if there is a message in the Mailbox
  if (Mailbox.messageAvailable()) {
    // read all the messages present in the queue
    while (Mailbox.messageAvailable()) {
      Mailbox.readMessage(message);
      SerialUSB.println(message);
    }

    SerialUSB.println("Waiting 10 seconds before checking the Mailbox again");
  }
  // wait 30 seconds
  delay(10000);
}

I’m not totally sure when I’d use a mailbox. The mailbox is a generic message and it’s not queued (only 1 message) so I think that using the standard bridging of values with get/put is more useful.

Final Comments

The Yun is missing a lot of networking libraries that are available with the ESP-8266 and ESP32 family of Arduino modules. In some cases like with MQTT that was a good Linux command line tool that could be used, but in cases like where you want to connect to a Redis server, you might have some challenges.

I really liked Yun’s built in uhttpd web server, this is far superior to the ESP-8266 Arduino web server library calls.

I found that putting logic into a combination of Python and Arduino could be a little confusing, but for projects with a lot of text or json data using Python would be a real plus. Also for projects where multitasking is required use the Linux process would be ideal.

Simple Terminal Interfaces

Typically our interfaces for projects use colorful web pages or custom GUIs. However there are many cases where a simple text interface is all that is required. This is especially true for SSH or remote connections from a Window’s client into a Raspberry Pi or Linux server.

In this blog I’d like to review a 1980’s technology called curses, with three examples. The first example will be simulated Rasp Pi scanning app in “C” and Python. The second and third examples will be in Python and they will show large text presentation and dynamic bars.

Python Curses

Python curses are standard in Python, and they include features such as:

  • support ASCII draw characters
  • basic color support
  • window and pad objects which can be written to and cleared independently

As a first example I wanted to have a colored background, a header and footer and some dynamic text.

curses_text

The first step is to define a curses main screen object (stdscr). The next step is to enable color and to create some color pairs. Using color pairs and the screen size (height, width = stdscr.getmaxyx()) it is possible to add a header and footer strip using the srtdscr.addstr command.

The stdscr.nodelay command allow the program to cycle until the stdscr.getch() call returns a key.

# curses_text.py - create a curses app with 10 dynamic values
#
import curses , time, random

# create a curses object
stdscr = curses.initscr()
height, width = stdscr.getmaxyx() # get the window size

# define two color pairs, 1- header/footer , 2 - dynamic text, 3 - background
curses.start_color()
curses.init_pair(1, curses.COLOR_RED, curses.COLOR_WHITE)
curses.init_pair(2, curses.COLOR_GREEN, curses.COLOR_BLACK)
curses.init_pair(3, curses.COLOR_WHITE, curses.COLOR_BLUE)

# Write a header and footer, first write colored strip, then write text
stdscr.bkgd(curses.color_pair(3))
stdscr.addstr(0, 0, " " * width,curses.color_pair(1) )
stdscr.addstr(height-1, 0, " " * (width - 1),curses.color_pair(1) )
stdscr.addstr(0, 0, " Curses Dynamic Text Example" ,curses.color_pair(1) )
stdscr.addstr(height-1, 0, " Key Commands : q - to quit " ,curses.color_pair(1) )
stdscr.addstr(3, 5, "RASPBERRY PI SIMULATED SENSOR VALUES" ,curses.A_BOLD )
stdscr.refresh()

# Cycle to update text. Enter a 'q' to quit
k = 0
stdscr.nodelay(1)
while (k != ord('q')):
# write 10 lines text with a label and then some random numbers
for i in range(1,11):
    stdscr.addstr(4+ i, 5, "Sensor " + str(i) + " : " ,curses.A_BOLD )
    stdscr.addstr(4+ i, 20, str(random.randint(10,99)) ,curses.color_pair(2) )
    time.sleep(2)
    k = stdscr.getch()

curses.endwin()

The simulated Pi values will refresh every  10 seconds until the “q” key is pushed and then the terminal setting are returned to normal (curses.endwin()) and the program exits.

“C” Curses Example

For this “C” example I used a Raspberry Pi. The curses library needs to be installed by:

 sudo apt-get install libncurses5-dev

The curses syntax is similar between “C” and Python but not 100%. For example in Python the addstr command includes a color pair reference, but in “C” this is not supported so an attribute on/off (attron/attroff) command is used to reference the color pair. Below is the “C” code:

/* c1.c - Basic Curses Example */

#include <curses.h>
#include <stdlib.h>
#include <unistd.h>

int main(void)
{
    int row, col, k;    
// Create a curses object and define color pairs
    initscr();
    getmaxyx(stdscr,row,col);
    start_color();
    init_pair(1,COLOR_RED,COLOR_WHITE);
    init_pair(2,COLOR_GREEN,COLOR_BLACK);
    init_pair(3,COLOR_WHITE,COLOR_BLUE);
    curs_set(0);
    noecho();
    //keypad(stdscr,TRUE);
    nodelay(stdscr, TRUE);
// Write a header and footer, first write colored strip, then write text
    bkgd(COLOR_PAIR(3));
    attron(COLOR_PAIR(1));
// Create a top and bottom color strip
    for (int i = 0; i < col; i++) {
        mvaddstr(0, i,  " ");
        mvaddstr(row-1, i,  " ");
    }
    mvaddstr(0, 0,  " Curses C Dynamic Text Example");
    mvaddstr(row-1, 0,  " Key Commands: q - to quit");
    attroff(COLOR_PAIR(1));   
    mvaddstr(2, 5,"RASPBERRY PI SIMULATED SENSOR VALUES" );
    refresh();
// Cycle with new values every 2 seconds until a q key (133) is entered    
    while (k != 113)
    {
        attroff(COLOR_PAIR(2));
        for (int i = 0; i < 10; i++) {
            mvprintw((4+i), 5,  " Sensor %d : ",i);
        }
        attron(COLOR_PAIR(2));
        for (int i = 0; i < 10; i++) {
            mvprintw((4+i), 20,  "%d",rand() %100);
        }
        k = getch();
        sleep(2);
    }
    endwin();
    exit(0);
}

To compile and run the program (c1.c) enter:

gcc -o c1 c1.c -lncurses
./c1

The “C” example should look very similar to the earlier Python example.

Figlet for Large Custom Text

Large Custom Text can be generated using the Python Figlet library.  Figlet has an extensive selection of text presentations and it uses standard ASCII character to generate the large text presentations. The Figlet library is installed by:

pip install pyfiglet

An example from the Python shell:

pyshell_figlet

For a Figlet example, I wanted to create a large heading and a large dynamic value.

curses_di

The Figlet library can be used to generate a string with user defined texted presented a large text-like format. A little bit of testing is required because the Figlet generated text can be 3,4,5 or more characters tall and the string needs to be added to very left end of the window.

# curses_di.py - show a large heading and large dynamic value
#
import curses, time
import pyfiglet, random

def get_io():
    global value1
    testvalue = str(random.randint(100,1000)/10) + " C"
    value1 = pyfiglet.figlet_format(testvalue, font = "starwars" )

# Create a string of text based on the Figlet font object
title = pyfiglet.figlet_format("Raspi Data", font = "small" ) 

stdscr = curses.initscr() # create a curses object
# Create a couple of color definitions
curses.start_color()
curses.init_pair(1, curses.COLOR_YELLOW, curses.COLOR_BLACK)
curses.init_pair(2, curses.COLOR_GREEN, curses.COLOR_BLACK)

# Write the BIG TITLE text string
stdscr.addstr(1,0, title,curses.color_pair(1) )
stdscr.addstr(8,0, "Sensor 1: GPIO 7 Temperature Reading" ,curses.A_BOLD)

# Cycle getting new data, enter a 'q' to quit
stdscr.nodelay(1)
k = 0
while (k != ord('q')):
    get_io() # get the data values
    stdscr.addstr(10,0, value1,curses.color_pair(2) )
    stdscr.refresh()
    time.sleep(2)

    k = stdscr.getch()

curses.endwin()

I found that the the small and doom fonts worked well in my testing. To check out and test Figlet fonts online see:

http://patorjk.com/software/taag/#p=display&f=Slant&t=Dude%20what%20are%20you%20doing%20%3F

Curses Windows

By defining a curses window it is possible to clear and write to a window that it is independent from the background. The syntax to create a curses window object is:

mynewwindow = curses.newwin(height, width, begin_y, begin_x)

Windows are ideal for applications where multiple items such as Figlet objects are used. Below is an example with two large Figlet values.

Figlet2win


# Create a static 2 large values example
#
import curses, time
import pyfiglet, random
# Create a string of text based on the Figlet font object
title = pyfiglet.figlet_format("Weather Station 2", font = "small" )

stdscr = curses.initscr() # create a curses object
# Create a couple of color definitions
curses.start_color()
curses.init_pair(1, curses.COLOR_YELLOW, curses.COLOR_BLACK)
curses.init_pair(2, curses.COLOR_GREEN, curses.COLOR_BLACK)

# Write the BIG TITLE text string
stdscr.addstr(1,0, title,curses.color_pair(1) )
stdscr.refresh()

win1 = curses.newwin(9, 44, 6, 4)
win1.addstr(8,0, "Sensor 1: Temperature Reading" ,curses.A_BOLD)

win2 = curses.newwin(9, 44, 6, 50)
win2.addstr(8,0, "Sensor 2: Humidity Reading" ,curses.A_BOLD)
value1 = pyfiglet.figlet_format("23 C", font = "doom" )
win1.addstr(0,0,value1,curses.color_pair(2) )
win1.refresh()
value2 = pyfiglet.figlet_format("35 %", font = "doom" )
win2.addstr(0,0, value2 ,curses.color_pair(2) )
win2.refresh()

# Hit any key to exit
stdscr.getch()
curses.endwin()

Dynamic Bars Example

For the Dynamic bars example I created a get_io function to simulate two real time data  values.

As a first step I created some background information such as headings, a header and a footer. By using the call: height, width = stdscr.getmaxyx() , I am able to position banners at the top and bottom of the terminal window. All of the background info is written to the stdscr object.

Two windows objects (win1 and win2) are used for the real time dynamic bars. Old bar data is removed using the win1.clear() and win2.clear() calls. Like the static example the dynamic bars are created by writing a fill character multiplied by the actual real time value (win1.addstr(1, 1, bar * value1) ). A window.refresh() command is used to show the changes.

The stdscr.getch() method is used to catch keyboard input, and the terminal program is exited when a quit character, “q” is entered.

The complete two dynamic bar program is shown below:


# Simple bar value interface
#
import curses
import time

# get_io is using random values, but a real I/O handler would be here
def get_io():
    import random
    global value1, value2
    value1 = random.randint(1,30)
    value2 = random.randint(1,30)

bar = '█' # an extended ASCII 'fill' character
stdscr = curses.initscr()
height, width = stdscr.getmaxyx() # get the window size
curses.start_color()
curses.init_pair(1, curses.COLOR_RED, curses.COLOR_WHITE)
curses.init_pair(2, curses.COLOR_GREEN, curses.COLOR_BLACK)
curses.init_pair(3, curses.COLOR_YELLOW, curses.COLOR_BLACK)

# layout the header and footer
stdscr.addstr(1,1, " " * (width -2),curses.color_pair(1) )
stdscr.addstr(1,15, "Raspberry Pi I/O",curses.color_pair(1) )
stdscr.hline(2,1,"_",width)
stdscr.addstr(height -1,1, " " * (width -2),curses.color_pair(1) )
stdscr.addstr(height -1,5, "Hit q to quit",curses.color_pair(1) )

# add some labels
stdscr.addstr(4,1, "Pi Sensor 1 :")
stdscr.addstr(8,1, "Pi Sensor 2 :")

# Define windows to be used for bar charts
win1 = curses.newwin(3, 32, 3, 15) # curses.newwin(height, width, begin_y, begin_x)
win2 = curses.newwin(3, 32, 7, 15) # curses.newwin(height, width, begin_y, begin_x)

# Use the 'q' key to quit
k = 0
while (k != ord('q')):
    get_io() # get the data values
    win1.clear()
    win1.border(0)
    win2.clear()
    win2.border(0)
# create bars bases on the returned values
    win1.addstr(1, 1, bar * value1, curses.color_pair(2))
    win1.refresh()
    win2.addstr(1, 1, bar * value2 , curses.color_pair(3))
    win2.refresh()
# add numeric values beside the bars
    stdscr.addstr(4,50, str(value1) + " Deg ",curses.A_BOLD )
    stdscr.addstr(8,50, str(value2) + " Deg ",curses.A_BOLD )
    stdscr.refresh()
    time.sleep(2)
    stdscr.nodelay(1)
    k = stdscr.getch() # look for a keyboard input, but don't wait

curses.endwin() # restore the terminal settings back to the original

cbars

For testing I used a random simulator for the data but the get_io function could be easily configured to connect to a Raspberry Pi or Arduino module.

The outline boxes in the window object could look strange if you are using a Window’s based SSH client like Putty. To create the problem in Putty’s settings, select: Window ->  Translations and use VSCII as the remote character set.

putty

Final Comments

Curses is definitely an ‘old school’ technology but it offers a simple solution for SSH and terminal based connections.

Pi with Neopixels/Simulated Neopixels

Neopixels are addressable full-colour RGB LEDs that come in a variety of different arrangements. Ranging from single LEDs, to matrix arrays and a variety of sewable components that can be used on wearable products.

dif_neopixels

Neopixels were originally made available for Arduino projects, but now there are also Python libraries for Raspberry Pi’s.

In this blog I will be looking at setting up neopixels components on Raspberry Pi’s, and then I will show some “soft” neopixel layouts using the Python Tkinter graphic library.

Getting Started

To load the Raspberry Pi neopixel libary comes from the nice people at Adafruit, and it is loaded by:

sudo pip3 install rpi_ws281x adafruit-circuitpython-neopixel

It is important to note that neopixels can draw a lot of power so consider looking at using external 5V power for projects with a lot of LEDs. The LED power consumption is based on:

  • How many neopixel LEDs are lit at one time, and
  • What the intensity of the LEDs is.

A few other import points are:

  • not all neopixel strips are the same. Different strips will vary greatly from the LED intensity, and more importantly on the RGB vs. GRB addressing.
  • NeoPixels must be connected to D10, D12, D18 or D21 to work
  • For NeoPixels to work on Raspberry Pi, you must run the code as root

The neopixels are wired with 3 pins : 5V (VCC), GND and signal in. The default signal in wires to Pi pin 18. Neopixel component can be connected in series with data in and out connectors.

data_in_out

Below is an example that will set all the LEDs to a light magenta and then it will cycle one LED to a brighter RED. The overall neopixel string has a 10% brightness.


# Python neopixel example to cycle an LEDs

import board
import neopixel
import time

ORDER = neopixel.RGB  # or neopixel.GRB
numpixels = 12

# Create a pixel object with 12 pixels and low intensity
pixels = neopixel.NeoPixel(board.D18,numpixels , brightness=0.10, auto_write=True, pixel_order=ORDER)
while True:
for i in range(numpixels):
   pixels.fill((10, 0, 40)) # fill all pixels in light magenta
   pixels[i] = (80,0,0) # fill one pixel in brighter red
   time.sleep(1)

Depending on the type and manufacturer of the neopixels the result will look a little different. Some trial and error testing will be required to determine if the strips are RGB or GRB.

 

Simulated Neopixels

If you don’t have neopixels or if what to simulate neopixels then the Python Tkinter graphic library can be used to create a variety of different arrangements. For my testing I create two arrangements: a strip and a matrix.

The important things that I learned was how to create a array object that could simulate the neopixel object. To do this in Python:


import tkinter as tk

root = tk.Tk()
root.title("Soft NeoPixel Strip")

numleds = 25

# Create an array that can be used later in Tkinter

ledstrip = ['' for i in range(numleds)]

for i in range(numleds):
   ledstrip[i] = tk.Label(root,relief='raised',width=3 ) # a label array
   ledstrip[i].grid(row = 0, column = i) # position the labels is a horizontal row
root.mainloop()

Simulated Strip Neopixel

Below is an example of a soft “strip” neopixel application with a demo function.

py_neo_Strip

# Python Neopixel Single Strip Presentation
#
import tkinter as tk

numleds = 25

theled = 0

def stringdemo():
    # move a coloured LED around the string
    global theled
    ledstrip[theled].configure(background= 'white')
    theled = theled + 1
    if theled >= numleds:
        theled = 0
    ledstrip[theled].configure(background= 'sky blue')
    root.after(500, stringdemo)
    
root = tk.Tk()
root.title("Soft NeoPixel Strip")

# create an LED object 
ledstrip = ['' for i in range(numleds)]

# put the LED object into a horizontal strip
for i in range(numleds):
    ledstrip[i] = tk.Label(root,width=2,height=1,relief='raised',background = 'white')
    ledstrip[i].grid(row = 0, column = (i+1))

root.after(500, stringdemo) #start a demo

root.mainloop()

Simulated Matrix Neopixels

Below is an example of a soft matrix neopixel application.

py_neo_Matrix

# Python Neopixel Matrix Presentation
#
import tkinter as tk

numleds = 100
rowcnt = 10
colcnt = int (numleds/rowcnt)

theled = 0

def stringdemo():
    # move a coloured LED around the string
    global theled
    ledstrip[theled].configure(background= 'dark gray')
    theled = theled + 1
    if theled >= numleds:
        theled = 0
    ledstrip[theled].configure(background= 'red')
    root.after(500, stringdemo)
    
    
root = tk.Tk()
root.title("Soft NeoPixel Matrix")

# create LED object
ledstrip = ['' for i in range(numleds)]

# put the LED object into a grid
for i in range(rowcnt):
    for j in range(colcnt):
        ledstrip[theled] = tk.Label(root,width=4,height=2,relief='raised',background = 'dark gray')
        ledstrip[theled].grid(row = i, column = j)
        theled = theled + 1

theled = 0 #reset the led index for the demo
root.after(500, stringdemo)

root.mainloop()

Summary

Neopixels can be used on custom lighting applicatons, for example I used them on a water fountain project.

Given a choice I would recommend using Arduino hardware over the Raspberry Pi hardware for neopixel projects. I found that the Arduino neopixel library to be much more stable and considerably faster than the Pi version.

Small Standalone Databases for IoT

For many IoT projects you want to historically save your data, however if you’re only saving a small amount of data using a database server such as MySQL can be overkill.

I wanted to come up with a way to save my data and be able to port it between projects and hardware.  My goal was to use Python because it has the best Raspberry Pi support. I looked at some options like:

  • CSV files – simple but filtering and sorting isn’t so easy
  • dBase files – good “old school” approach, however it’s no longer support in Excel and MS Access so viewing the data offline is a little ugly
  • Python Pickle and PickleDB – great for saving game scores but no real sorting
  • SQLite – excellent light weight database

For my light weight Python applications I was really happy with TinyDB. TinyDB offers:

  • a JSON storage interface, so you could easily migrate to MongoDB (or equivalent)
  • pure Python with no dependencies,  (so it runs on all OS’s and versions of Python)
  • supports queries and searches

Getting Started with TinyDB

To install TinyDB :

pip install tinydb

TinyDB is standalone so there are no server components. Your Python code talks directly to one of more JSON files, or you can also create multiple tables in one JSON.

Below is a Python example that create a JSON file with two tables.


# Use TinyDB to create a JSON file with 2 tables
#
from tinydb import TinyDB, Query

db = TinyDB('db2.json')
db.purge_tables() # Clear out the file's old data

table = db.table('Moisture')

table.insert({"date":"2019-05-06","time":"16:00","moisture":200})
table.insert({"date":"2019-05-06","time":"16:10","moisture":210})
table.insert({"date":"2019-05-06","time":"16:20","moisture":220})

table2 = db.table('Light')

table2.insert({"date":"2019-06-06","time":"16:00","light":30})
table2.insert({"date":"2019-06-06","time":"16:10","light":35})
table2.insert({"date":"2019-06-06","time":"16:20","light":36})

db.close()

#
import json

with open('db2.json') as f:
data = json.load(f)

print("JSON file with 2 tables\n")
print(json.dumps(data, indent = 4, sort_keys=True))

The JSON file will have a format like:

json_2tables

A Test Example

For a test example I wanted to simulate a month’s worth of 5 minute data. Then I would try some queries to show the data.

 
# Create a month's worth of 10 minute simulated JSON data
#
# Use a format of:
#    "Pi_1": {
#        "1": {
#            "date": "2019-04-01",
#            "time": "01:00",
#            "moisture": 577,
#            "light": 30
#        }

from tinydb import TinyDB, Query
import random

db = TinyDB('/writing/blog/tinydb/db1.json')
db.purge_tables()
table = db.table('Pi_1')

for iday in range(1,31):
    print("day: ",iday)
    for ihour in range(0,24):
        print("\thour: ", ihour)
        for imin in range(0,56,5):
            thedate = "2019-04-{:02}".format(iday)
            thetime = "{:02}:{:02}".format(ihour, imin)
            table.insert({"date": thedate,"time": thetime ,"moisture": random.randint(200,600),"light":  random.randint(20,100)})


print ("Finished updating JSON file\n\n")
db.close(

Once the JSON data file was created I used a simple Python Tkinter application to query the data by day. My goal was to see how easy it was to query the data and return results. Below is a my TinyDB Tkinter viewing application and code. The bottom slider is used to select the day in the month.

dbtiny_data


# Use a Tkinter 
# Create a dialog that uses TinyDB to query 1 day of data
# from a JSON file
#
from tinydb import TinyDB, Query
import tinydb 
import tkinter as tk

db = TinyDB('db1.json')

# get a selected days data
def sel():
   thedate = "2019-04-{:02}".format(int(var.get()))
   print(thedate)
   table = db.table('Pi_1') 
   data1 = table.search(Query()['date'] == thedate)
   print(data1)
   if data_list.size() > 0:
       data_list.delete(1,data_list.size() -1 )
   for items in data1:
       data_list.insert(data_list.size()+ 1,
                        items['date'] + " " + items['time'] +
                        " -  Moisture : " + str(items['moisture']) +
                        " -  Light : " + str(items['light']) )


root = tk.Tk()
root.title("DBtiny - Daily Data")

var = tk.DoubleVar()
data_list = tk.Listbox(root,height=20, width= 60)
data_list.grid(row=1,column=1)
                     
day_scale = tk.Scale( root,variable = var, from_=1,to=30,orient='horizontal',length=300 ).grid(row=2,column=1)

button1 = tk.Button(root, text="Get Daily Values", command=sel).grid(row=3,column=1)

root.mainloop()

Summary

TinyDB is a good solution for small IoT projects where I’m using Python and I’m not collecting a lot of data.

My next step would be to look at presenting the JSON data in Node-Red.