GnuPlot: Realtime Plots in 20 lines

There are some excellent charting and plotting packages but if you’re like me you sometimes want to do a quick test plot to capture some realtime data.

Gnuplot has been around for quite awhile and I was happily surprised at what it can do. I was amazed that I could make real-time bar and line charts in only 20 lines of scripting code. In this blog I’ll introduce Gnuplot and show two examples. The first will show the status Raspberry Pi I/O pins, and the second example will be a line chart of CPU diagnostics.

Getting Started

Gnuplot can be installed on Linux, Windows, and Mac. To install Gnuplot on Ubuntu:

sudo apt-get install gnuplot

Gnuplot is typically run as a command line utility, but it can also be run manually. When Gnuplot is run manually the charting instructions and data values can be inserted directly. Below is an example where 4 sets of data points are plotted in a line chart.

pi@raspberrypi: $ gnuplot

gnuplot> $Mydata << EOD
# Now enter some data                                
2 1
3 1.5
4 2.1
5 3.3
EOD
gnuplot> plot $Mydata with line

Data block names must begin with a $ character, which distinguishes them from other types of persistent variables. The end-of-data delimiter (EOD in the example) may be any sequence of characters. For this example the plot command will use the data in the $Mydata variable and create a line chart.

A Static Bar Chart

A simple but useful example of Gnuplot would be to show the realtime status of the Raspberry Pi General Purpose Input/Output (GPIO) pins as a bar chart.

A bar chart presentation can be created using a data file of:

# gpio.dat - data file for GPIO pin values
# column1 = chart position, column2 = heading, column3 = value
0 GPIO2 0
1 GPI03 1
2 GPI04 1
#....

To plot a bar chart the fill style and bar width needs to be defined. The plot using 1:3:xtic(2) argument will make the first column in the data file the x-position, the third column the y-value and the x-labels be the second column. The interactive commands to plot the file:

pi@raspberrypi:~/pete/gnuplot $ gnuplot 

gnuplot> set style fill solid
gnuplot> set boxwidth 0.5
gnuplot> plot "gpio.dat" using 1:3:xtic(2) with boxes title ""

Real-Time Bar Chart of PI GPIO

The previous example used a manually created gpio.dat data file. The status of GPIO pins can be found using the gpio command line utility. To get the status of GPIO pin 9:

gpio read 9

By adding some Bash and awk script it is possible to create a gpio.dat file:

$ gpio read 9
1
$ gpio read 9 | awk '{ print "9 GPIO9 " $1 }'
9 GPIO9 1
$ gpio read 9 | awk '{ print "9 GPIO9 " $1 }' > gpio.dat
$ cat gpio.dat
9 GPIO9 1

To make a dynamic bar chart I created a Gnuplot script file (gpio_bars.txt). The Gnuplot scripting language is quite powerful, and it support a wide range of functions and control statements.

Rather than manually adding lines for each GPIO pin status a for loop can iterate from pins 2-29. A system command is used to run the gpio utility and bash commands. To refresh the data a replot and a pause command is used.

# Create Dynamic bar that read GPIO pins every 5 seconds
#
set title "PI GPIO Data"
set boxwidth 0.5
set style fill solid

# Create a dummy file to get started
system "echo '0 GPIO2 1' > gpio.dat"

plot "gpio.dat" using 1:3:xtic(2) with boxes title ""

while (1) {  # make a new 'gpio.dat' every cycle with fresh data
  system "echo '' > gpio.dat"
  do for [i=2:29] {
    j = i-2 # put first GPIO pin at position 0
    system "gpio read " .i.  "  | awk '{ print  \"" . j . " GPIO" . i . " \" $1 }' >> gpio.dat
  }
  replot
  pause 5
}

To run this script enter:

gnuplot -persist gpio_bars.txt

A Simple Line Chart

A bar chart presentation can be created using a data file of:

# GPU.dat - a time stamp with two data points
18:48:30 51.0 49.0
18:48:40 50.5 49.5
18:48:45 51.5 49.0
18:48:50 50.0 50.5
18:48:55 50.5 49.5

The interactive Gnuplot commands to show a line chart of this data would be:

pi@raspberrypi: $ gnuplot

gnuplot> set xdata time
gnuplot> set timefmt "%H:%M:%S"
gnuplot> set format x "%H:%M:%S"
gnuplot> plot "gpu.dat" using 1:2 with line title "GPU temp" ,\
>"gpu.dat" using 1:3:3 with line title "CPU temp"  

A few extra lines are needed in the Gnuplot script. First the plot needs to know that the x-axis is time data, and it needs to know the format of the time data. Multiple data points can be plotted at the same time, and the using argument tells Gnuplot the x:y-point data. (If the data file had a third point the using reference would be 1:4:4 ).

A Real-Time Line Chart

Linux has a lot of useful command line trouble shooting tools, one of these is the sensors utility that allows users to get fan speed and temperatures.

$ sensors
dell_smm-virtual-0
Adapter: Virtual device
Processor Fan: 2676 RPM
CPU: +47.0°C
Ambient: +38.0°C
SODIMM: +37.0°C
...

Using some Bash and Awk commands it is possible to get the fan speed and CPU temperature:

$ sensors | grep RPM
Processor Fan: 2685 RPM
$ sensors | grep RPM | awk '{print $3}'
2685

$ sensors | grep CPU
CPU: +50.0°C
$ sensors | grep CPU | awk '{print $2}'
+48.0°C
$ sensors | grep CPU | awk '{print substr($2,2,4)}'
48.0

Awk supports a systime() call to return the present date/time, and a strftime() call to customize the presentation of the time/date. (Note: a Raspberry Pi might need to have gawk installed to get this functionality, use “sudo apt-get install gawk”).

The next step is to format the sensor output with a timestamp:

$ sensors |grep RPM | awk '{print strftime("%H:%M:%S ", systime()) $3}'

10:26:46 2687

$sensors | grep CPU | awk '{print strftime(\"%H:%M:%S \",systime()) substr($2,2,4)}'

10:27:46 49.0

Now that a time and value string can be generated a Gnuplot script can be created (line_fan_cpu.txt) to show real time data.

To make the Bash code a little easier two data files are created, fan.dat and cpu.dat.

The plot has to account for different scale ranges, so y2range and y2label definitions are used. The final script addition is to include an axes (x1y2 or x1y2) to each plot point, this lines up the data value to the right or left y-axis.

The complete Gnuplot script to show fan speed and CPU temperature is only 20 lines of code!

# Create a Plot or User and System CPU Usage, update every 5 seconds
#
set title "GnuPlot - Fan Speed  and CPU Temperature"
set yrange [2650:2700] 
set ylabel "Fan Speed"
set y2range [43:49] 
set y2label "CPU Temp (C)"
set y2tics
set xdata time
set timefmt "%H:%M:%S"
set format x "%H:%M:%S"

system "sensors |grep RPM | awk '{print strftime(\"%H:%M:%S \", systime()) $3}' > fan.dat"
system "sensors | grep CPU | awk '{print strftime(\"%H:%M:%S \",systime()) substr($2,2,4)}'  > cpu.dat"

plot "fan.dat" using 1:2  with lines axes x1y1 title "fan speed (RPM)",  "cpu.dat" using 1:2 with lines axes x1y2 title "CPU Temp (C)"
while (1) {
	pause 5
	system "sensors |grep RPM | awk '{print strftime(\"%H:%M:%S \", systime()) $3}' >> fan.dat"
	system "sensors | grep CPU | awk '{print strftime(\"%H:%M:%S \", systime()) substr($2,2,4)}'  >> cpu.dat"
	replot
}

To run this script enter: $ gnuplot -persist line_fan_cpu.txt

Final Comments

I won’t give up using plotting packages like MatPlotlib or ggplot, but I was very impressed how easy it was to create real-time plots using Gnuplot.

Manipulating the Bash/awk script can be a little complex but it’s incredible useful to be able to use output from almost any command line utility in Gnuplot.

Gnuplot can plot a very large number of data points, but it makes sense to a tail command to only show the latest x-number of points.

Animated Node-Red Graphics with MQTT and SVG

There is a new SVG (Scalar Vector Graphics) node that is available for Node-Red  Dashboards. This node allows for animated Node-Red graphics that can be viewed on a smart phone.

In this blog I wanted to document an example of integrating MQTT messaging to SVG animated graphics.

Getting Started

If you’re unfamiliar with SVG graphics that are some good tutorials. For my own reference I wrote some notes on SVG and Javascript integration.

The Web Dashboards should be at the latest version,  you can do this in the “Manage Palette” option. The node-red-contrib-ui-svg node can be install from the “Manage Palette” option, or manually by:

cd ~/.node-red)
npm install node-red-contrib-ui-svg

There is some excellent documentation on this node.

The SVG Graphic node has a built-in SVG graphic editor or SVG code can be pasted directly into the “SVG Source” tab.

svg_source_edit

For my project I used industrial SVG examples from: https://www.opto22.com/support/resources-tools/demos/svg-image-library

The SVG editor is useful for identified and defining SVG items that are you’d like to animate.

svg_editor_ids

For this project the solar panels (id=panels) and the output (id=watts) were to be dynamically updated from MQTT.

SVG items can  be dynamically update by: 1)  the “Input Bind” tab in the node’s definition or, 2) as an input message.

An example using the input message approach would be:

In this example the colour of the panels is set to green using an injector and function node.

 

MQTT with SVG Graphics

There are some good MQTT brokers, such as Mosquitto that can be used. Node-Red also has a MQTT broker node (MOSCA) that is easy to install.

The earlier test logic can be adapted to connect MQTT inputs:

svg_mqtt_example

For this example two MQTT tags were used: 1) watts, and 2) panel_status.

MQTT testing can be done with the MQTT client tools, that are installed by:

 sudo apt-get install mosquitto mosquitto-clients -y

From a terminal MQTT tags can be published to a broker  (-h servername) with a topic (-t thetopic) and a message (-m themessage) :

mosquitto_pub -h 192.168.0.111 -t watts -m "123"
mosquitto_pub -h 192.168.0.111 -t panel_status -m "gold"

In Node-Red the watts text  is updated by the function node code of:


// Pass the MQTT payload
// and update the text, hardcode the units
//

msg.payload = {
"command": "update_text",
"selector": "#watts",
"textContent": (msg.payload + " C")
}

return msg;

The panels color is changed by the function node code of:

//
// Pass the MQTT payload
// as a fill color attribute
//

msg.payload = {
"command": "update_style",
"selector": "#panels",
"attributeName": "fill",
"attributeValue": msg.payload
}

return msg;

Final Comments

Drawing SVG graphics from scratch is awkward, but there are some great Internet examples of pre-build SVG graphics. All it takes is a bit of time to find the graphic items that you need to animate.

In this blog I looked at making SVG files dynamic. It is also possible to put “hot links” on the SVG file to call URLs or to send messages back to Node-Red.

 

 

Home Assistant with Node-Red

Home Assistant is an open source home automation platform that can monitor and control smart home devices and it integrates with many of other common systems.

HA_demo

Home Assistant installation is targeted for Raspberry Pi’s but other hardware options are available.

I was very impressed how easy it was to install Home Assistant and get a basic home integration system up and running.

There is a huge number of integration solutions (1500+) that connect to most of the mainstream products. However if you want to do some custom Arduino or Raspberry Pi connections there isn’t an easy “out of the box” solution.  To solve this requirement Home Assistant has included Node-Red as an add-on.

Node-RED is a visual programming tool for wiring together hardware devices, APIs and online services.

I found that getting the Node-Red integration was a little tricky. This blog will show how to get Node-Red integration working and it includes a simple simulator circuit.

Getting Started

The installation instructions are very straightforward. I would recommend using a wired connection for your Raspberry Pi. A wireless network connection is 100% possible but it is not in the base installation directions.

After the basic installation is complete, add-ons can be installed under the Supervisor->Dashboard. I would recommend installing “File editor” and “Terminal & SSH” add-ons along with Node-Red.

ha_addons

I found that the Node-Red installed without any problems but it required some configuration changes before it would run.

In the Node Red add-on you will need to add a credential_secret and a password.

nodered_config

If Node-Red doesn’t start look at the log for errors (it’s at the bottom of the same page).

nodered_log

The base Node-Red installation has a very good selection of pre-installed nodes. If you wish to add more nodes see the “Manage Pallet” option that is accessed from the top right options icon.

At this stage Node-Red is somewhat standalone and it is not fully integrated with Home Assistant.

Integrating Node-Red with Home Assistant

The directions and files for Node-Red integration  can be downloaded to your PC.

ha_nr_int_files

Specifically you want to custom_components/nodered directory and files, which will need to be moved to the Raspberry Pi. The Home Assistant “File editor” add-on can be used to create Pi directories and move files from your PC.

nodered_files

The following directories and file should now exist:

/root/config/custom_components/nodered/__init__.py
/root/config/custom_components/nodered/__pycache__
/root/config/custom_components/nodered/binary_sensor.py
/root/config/custom_components/nodered/config_flow.py
/root/config/custom_components/nodered/const.py
/root/config/custom_components/nodered/discovery.py
/root/config/custom_components/nodered/manifest.json
/root/config/custom_components/nodered/save.txt
/root/config/custom_components/nodered/sensor.py
/root/config/custom_components/nodered/services.yaml
/root/config/custom_components/nodered/switch.py
/root/config/custom_components/nodered/websocket.py

/root/config/custom_components/nodered/.translations/en.json

Once this is complete Home Assistant will need to be restarted.

Including Node-Red Integrations

The next step is to create sensors and switches in Node-Red that can be accessed in Home Assistant. Below is a simple circuit that sends a random number (0-100) to a HA entity.

This logic uses a Big Timer node, that generates a pulse every minute from the middle output pin. An injector node allow you to force a new value. A random node will output a new random number whenever the Big Timer or Inject nodes are triggered.

nr_circuit

Double-click on the HA entity to configure the HA server and other properties.

nr_entity_config

Once the logic is complete click the “Deploy” button to make the logic active.

Node-Red integration is enabled by adding it in the Configuration->Integration page.

nr_new_int

nr_entities

Overview Dashboard with Node-Red Data

The final step is to modify the Overview Dashboard to include the Node-Red Entity.

For this example I added a gauge component using the Orange-Plus at the bottom right of this Configure UI page.

config_ui

On the live Overview page it is possible to click on the gauge card and get more information about this sensor.

HA_overview

Final Thoughts

Home Assistant is a very well structured home automation solution that offers a number of excellent approaches to bring in data.

Node-Red is a very flexible programming environment that help expands connectivity to Arduino, Raspberry Pi and other 3rd party services that are not in the base Home Assistant software.

For information on how to connect an Arduino module to Node-Red see:

https://funprojects.blog/2018/03/04/arduino-talking-mqtt-to-node-red/

https://funprojects.blog/2018/02/18/arduino-talking-to-node-red-and-python/

 

SVG Web Animation

There are a number of good Web animation techniques that are available. Flash animation support is being discontinued in 2020, but luckily there are some good alternatives, such as HTML 5 Canvas, Scalar Vector Graphics (SVG),  and even CSS (Cascading Style Sheets) can be used .

HTML 5 Canvas draws 2D graphics, on the fly with a JavaScript. The canvas is rendered pixel by pixel. In canvas, once the graphic is drawn, it is forgotten by the browser. If its position should be changed, the entire scene needs to be redrawn, including any objects that might have been covered by the graphic.

SVG is a language for describing 2D graphics in XML, which means that every element is available within the SVG DOM. In SVG, each drawn shape is remembered as an object. If attributes of an SVG object are changed, the browser can automatically re-render the shape.

In this blog I wanted to document my SVG testing.

Using SVG

SVG can be used inline within an HTML document. Below is an example with a rectangle, circle and some text.

<!DOCTYPE html>
<html>
<body>

<svg width="400" height="400">
  <rect width="100" height="100" fill= "blue" />
  <circle cx="150" cy="50" r="40" fill="green" />
  <text x="10" y="120" fill="red">SOME SVG TEXT</text>
</svg>

</body>
</html>

This will generate a web like:

svg1

An SVG element has a width and height property and all the drawing elements are positioned within this area.

Dynamic SVG Bar

A bar can be created using a rectangle () element. Javascript is used to dynamically adjust the top and height of the rectangle.

<!DOCTYPE html>
<html>
<head>
<script type="text/JavaScript">

setTimeout(updatebar, 2000);

function updatebar() {
// 
  var newvalue = (Math.random() * 300); // get a random integer 0-100
  if (newvalue > 250) {
    document.getElementById("bar1").style.fill = "red";
  } else {
    document.getElementById("bar1").style.fill = "green";
  }
  newvalue = Math.round(newvalue);
  var newheight = newvalue.toString();
  var num = 300 - newvalue - 5;
  var newy = num.toString();  

  document.getElementById("bar1").setAttribute("height",newheight);
  document.getElementById("bar1").setAttribute("y",newy);
  document.getElementById("bartext").innerHTML = "value: " + newheight ;
  setTimeout(updatebar, 2000);
}
</script>
</head>
<body>
<h2> SVG Bar Example</h2>
<svg width="400" height="400" >
  <rect x="0" y="0" width="160" height="300" style="fill:white;stroke:black;stroke-width:4;" />
  <rect id="bar1" x="5" y="145" width="150" height="150" style="fill:green;stroke:black;stroke-width:2;" /> 
  <text id="bartext" x=30 y = 320 style="font-family:verdana;font-weight:bold">value: 50 </text>
</svg>

</body>
</html>

svg_bar

Animate Graphics

There some SVG drawing packages such as: https://www.drawsvg.org/drawsvg.html

There are also some free libraries, I used: https://www.opto22.com/support/resources-tools/demos/svg-image-library to animate a flow valve.

I needed to do a little bit of trial and error but I was able to add a text element to the bottom of the SVG that could be animated with Javascript. Below is the SVG file:

<svg x="0px" y="0px" viewBox="0 0 100 100" xml:space="preserve"><svg><g>
<g id="shape">
  <circle fill="#999999" cx="50.093" cy="32.75" r="32.75"></circle>
  <rect x="0.176" y="71.667" fill="#999999" width="99.833" height="26.833"></rect>
  <rect x="44.926" y="63" fill="#999999" width="10.333" height="10.5"></rect>
  <rect y="70" fill="#999999" width="4.5" height="30"></rect> 
  <rect x="95.509" y="70" fill="#999999" width="4.5" height="30"></rect>
</g>
<g id="linear">
  <linearGradient class="linear" id="SVGID_1_" gradientUnits="userSpaceOnUse" x1="50.0926" y1="94.5113" x2="50.0926" y2="75.6554">
    <stop offset="0" style="stop-color:#E6E6E6"></stop>
    <stop offset="1" style="stop-color:#000000"></stop>
  </linearGradient>
  <polygon class="linear" fill="url(#SVGID_1_)" points="70.031,75.655 60.041,79.079 40.144,79.079 30.154,75.655 0.176,75.655 0.176,94.511 
    29.289,94.511 40.144,90.791 60.041,90.791 70.896,94.511 100.009,94.511 100.009,75.655   ">
</g>
<g id="light">
  <circle fill="#E6E6E6" cx="50.093" cy="32.75" r="29.074"></circle>
  <path fill="#E6E6E6" d="M44.833,71.5v-6.409c0.606,0.096,1.217,0.179,1.833,0.241V71.5H44.833z"></path>
</g>
<g id="hlight">
  <path fill="#FFFFFF" d="M18.343,33.75C18.343,15.663,33.005,1,51.093,1c8.789,0,16.763,3.469,22.646,9.104
    C67.777,3.881,59.391,0,50.093,0c-18.087,0-32.75,14.663-32.75,32.75c0,9.298,3.881,17.685,10.104,23.646
    C21.812,50.514,18.343,42.539,18.343,33.75z">
  <rect x="26.093" y="21.474" fill="#FFFFFF" width="48" height="23.053"></rect>
</g>
<g id="shadow">
  <path fill="#000000" d="M72.738,9.104c5.635,5.882,9.104,13.857,9.104,22.646c0,18.087-14.663,32.75-32.75,32.75
    c-8.789,0-16.763-3.469-22.646-9.104C32.408,61.619,40.795,65.5,50.093,65.5c18.087,0,32.75-14.663,32.75-32.75
    C82.843,23.452,78.961,15.066,72.738,9.104z">
  <path d="M55.259,71.5v-6.409c-0.606,0.096-1.217,0.179-1.833,0.241V71.5H55.259z"></path>
  <polygon points="27.235,22.589 74.093,22.589 74.093,21.474 26.093,21.474 26.093,44.526 27.235,44.526  "></polygon>
  <path d="M22.018,33.75c0-16.057,13.017-29.074,29.074-29.074c7.774,0,14.83,3.057,20.047,8.028
    c-5.296-5.558-12.764-9.028-21.047-9.028c-16.057,0-29.074,13.017-29.074,29.074c0,8.283,3.469,15.751,9.028,21.047
    C25.076,48.58,22.018,41.524,22.018,33.75z">
</g>
</g>
<text id="flowval" name="flowval" x = "40" y = "40" fill= "green" style="font-family:verdana;font-weight:bold"> 00 </text></svg>
</svg>

External SVG files can be used in a Web page by:

  •  <img – as an image file. (Only good for static or self-contained SVG files)
  • <frame – as a frame.
  • <object – as a object reference within an existing web document
  • <embed – as an embedded element

Unfortunately I found that Chrome only supports inline SVG with Javascript. Microsoft Edge and Firefox worked with both the object and embed tags.

Below is the code that I used to animate the text on the flow valve:

<!DOCTYPE html>
<html>
<head>
<script type="text/JavaScript">

setTimeout(updatebar, 2000);

function updatebar() {
// 
  var randvalue = (Math.random() * 100); // get a random integer 0-100
  var flowvol = Math.round(randvalue.toString());
  // set the text in the svg object   
  var svgobj = document.getElementById("flowmeter");
  var svgdoc =  svgobj.getSVGDocument();
  svgdoc.getElementById("flowval").textContent =  flowvol ;
  
  setTimeout(updatebar, 2000);
}
</script>
</head>
<body>
<h2> SVG Flow Meter</h2>
<embed id="flowmeter" height= '50%' width='100%' src="flowmeter.svg" >

</body>
</html>

flowmeter_1

Final Comments

I found that it can be a little challenging to tweek existing SVG library examples. The browser “development tools” can be used to find and test SVG elements. Below is an example where I found the solar panel element that I need to animate.

svg_devtools

For 100% cross-browser capability it would be recommended to put all the SVG code in one document.

The next step will be to use a Raspberry Pi Web app and some AJAX calls to animate the data.

Pi Charts with PySimpleGUI

There are a lot of good charting packages available for IoT and Rasperry Pi projects.

The PySimpleGUI python libary stands out in its ability to have the same code for both a local GUI and a Web interface. PySimpleGUI isn’t focused as a charting package but it has the canvas and graph elements that allow you to create real time bar charts and real time trend charts.

Getting Started with Graph Elements

For some background on PySimpleGUI see: PySimpleGUI – quick and easy interfaces

The graph element can have different co-ordinate orientations, for example the center can be (0,0) or the bottom left can be (0,0). A graph element is created with the syntax:

Graph(canvas_size, graph_bottom_left, graph_top_right …)

An example with 2 different co-ordinate orientations would be:


# A basic PySimpleGUI graph example
import PySimpleGUI as sg

bcols = ['blue','red','green']
myfont = "Ariel 18"

gtop = sg.Graph((200,200), (0,0),(200,200),background_color="white")
gcenter = sg.Graph((200,200), (-100,-100), (100,100),background_color="white")

layout = [[sg.Text('Graph: 0,0 at bottom left',font=myfont)],
[gtop],
[sg.Text('Graph: 0,0 at center',font=myfont)],
[gcenter],
[sg.Exit()]]

window = sg.Window('Graph Example', layout)

# Write text and lines
event, values = window.read(timeout=0)

gtop.draw_text(text="(0,0)", location=(0,0))
gcenter.draw_text(text="(0,0)", location=(0,0))

gtop.draw_text(text="(50,50)", location=(50,50))
gcenter.draw_text(text="(50,50)", location=(50,50))

gtop.draw_line((0,0),(50,50))
gcenter.draw_line((0,0),(50,50))

# Wait for a key to exit
window.read()

window.close()

sg_chart04

Bar Charts

Bar charts can be created using the graph.draw_rectangle() method. Below is an example that takes a command line input to toggle between a tkinter local interface and a web interface. This example has 3 input points that are scanned every 2 seconds.


import sys
import random

# Pass any command line argument for Web use
if len(sys.argv) &gt; 1: # if there is use the Web Interface
    import PySimpleGUIWeb as sg
    mode = "web"
    mysize = (20,2)
else: # default uses the tkinter GUI
    import PySimpleGUI as sg
    mode = "tkinter"
    mysize = (12,1)
BAR_WIDTH = 150
BAR_SPACING = 175
EDGE_OFFSET = 3
GRAPH_SIZE = (500,500)
DATA_SIZE = (500,500)

bcols = ['blue','red','green']
myfont = "Ariel 18"

#update with your ip
myip = '192.168.0.107'

graph = sg.Graph(GRAPH_SIZE, (0,0), DATA_SIZE)

layout = [[sg.Text('Pi Sensor Values',font=myfont)],
  [graph],
  [sg.Text('PI Tag 1',text_color=bcols[0],font=myfont,size= mysize ),
  sg.Text('PI Tag 2',text_color=bcols[1],font=myfont,size= mysize ),
  sg.Text('PI Tag 3',text_color=bcols[2],font=myfont,size= mysize)],
  [sg.Exit()]]

if mode == "web":
    window = sg.Window('Real Time Charts', layout,web_ip=myip, web_port = 8888, web_start_browser=False)
else:
    window = sg.Window('Real Time Charts', layout)
while True:
    event, values = window.read(timeout=2000)
    if event in (None, 'Exit'):
        break

    graph.erase()
    for i in range(3):
# Random value are used. Add interface to Pi sensors here:
        graph_value = random.randint(0, 400)
        graph.draw_rectangle(top_left=(i * BAR_SPACING + EDGE_OFFSET, graph_value),
        bottom_right=(i * BAR_SPACING + EDGE_OFFSET + BAR_WIDTH, 0), fill_color=bcols[i])
        <span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>graph.draw_text(text=str(graph_value), location=(i*BAR_SPACING+EDGE_OFFSET+15, graph_value+10),color=bcols[i],font=myfont)

window.close()

The presentation between the tkinter and Web interface is almost identical, but not 100% some tweeking on text sizing is required.

Real Time Trend Charts

It is important to note that the PySimpleGUIWeb is still in development so there may be some compatibility issues when trying to toggle between the tkinter and Web versions.

Below is an example that will create a realtime chart.

sg_realtimechart

#!/usr/bin/env python

import array
from datetime import datetime
import sys
import random

# Pass any command line argument for Web use
if len(sys.argv) > 1: # if there is use the Web Interface
    import PySimpleGUIWeb as sg
    mode = "web"
    mysize = (20,2)
else: # default uses the tkinter GUI
    import PySimpleGUI as sg
    mode = "tkinter"
    mysize = (12,1)

from threading import Thread

STEP_SIZE = 1  # can adjust for more data saved than shown
SAMPLES = 100 # number of point shown on the chart
SAMPLE_MAX = 100 # high limit of data points
CANVAS_SIZE = (1000, 600)
LABEL_SIZE = (1000,20)

# create an array of time and data value
pt_values = []
pt_times = []
for i in range(SAMPLES+1): 
    pt_values.append("")
    pt_times.append("")
    

def main():

    timebar = sg.Graph(LABEL_SIZE, (0, 0),(SAMPLES, 20), background_color='white', key='times')
    graph = sg.Graph(CANVAS_SIZE, (0, 0), (SAMPLES, SAMPLE_MAX),
          background_color='black', key='graph')

    layout = [
        [sg.Quit(button_color=('white', 'red')),sg.Button(button_text="Print Log", button_color=('white', 'green'),key="log"),
         sg.Text('     ',  key='output')],
        [graph],
        [timebar],       
    ]
    if mode == 'web':
        window = sg.Window('Pi Trend Chart', layout,
                       web_ip='192.168.0.107', web_port = 8888, web_start_browser=False)
    else:
        window = sg.Window('Pi Trend Chart', layout)

    graph = window['graph']
    output = window['output']

    i = 0
    prev_x, prev_y = 0, 0

    while True: # the Event Loop
  
        event, values = window.read(timeout=1000)
        if event in ('Quit', None):  # always give ths user a way out
            break
        if event in ('log'): # print the recorded time/data arrays
            print("\nReal Time Data\n")
            for j in range(SAMPLES+1):
                if pt_times[j] == "": #only print updated info
                    break
                print (pt_times[j], pt_values[j])

        # Get data point and time
        data_pt = random.randint(0, 100)
        now = datetime.now()
        now_time = now.strftime("%H:%M:%S")
        # update the point arrays
        pt_values[i] = data_pt       
        pt_times[i] = str(now_time) 

        window['output'].update(data_pt)
        
        if data_pt > SAMPLE_MAX:
            data_pt = SAMPLE_MAX
        new_x, new_y = i, data_pt

        if i >= SAMPLES:
            # shift graph over if full of data
            graph.move(-STEP_SIZE, 0)
            prev_x = prev_x - STEP_SIZE
            # shift the array data points
            for i in range(SAMPLES):
                pt_values[i] = pt_values[i+1]
                pt_times[i] = pt_times[i+1]
                    
        graph.draw_line((prev_x, prev_y), (new_x, new_y), color='red')
        
        prev_x, prev_y = new_x, new_y
        i += STEP_SIZE if i < SAMPLES else 0
    
        
        timebar.erase()
        # add a scrolling time value 
        time_x = i
        timebar.draw_text(text=str(now_time), location=((time_x - 2),7) )
        # add some extra times
        if i >= SAMPLES:
            timebar.draw_text(text=pt_times[int(SAMPLES * 0.25)], location=((int(time_x * 0.25) - 2),7) )
            timebar.draw_text(text=pt_times[int(SAMPLES * 0.5)], location=((int(time_x * 0.5) - 2),7) )
            timebar.draw_text(text=pt_times[int(SAMPLES * 0.75)], location=((int(time_x *0.75) - 2),7) )
        if i > 10:
            timebar.draw_text(text=pt_times[1], location=( 2,7) )
        
    window.close()


if __name__ == '__main__':
    main()

Final Comment

If you are looking at doing some charting and you want to have both a local and a web interface then PySimpleGUI and PySimpleGUIWeb will be something that you should take a look at.

PySimpleGUI – quick and easy interfaces

The Python PySimpleGUI project has two main goals, a simpler method for creating graphical user interfaces (GUIs), and common code for Tkinter, QT, xW and Web graphics.

I feel comfortable doing my own Python Tkinter and Web interfaces, but using common code for both local interfaces and Web apps could be extremely useful for Rasp Pi projects.

In the this blog I wanted to introduce PySimpleGUI by creating a local GUI/Web interface to control a Raspberry Pi Rover, all in less than 60 lines of code.

Getting Started with PySimpleGUI

The Python PySimpleGUI project has a number of “ports” or versions. The main version is for Tkinter based graphics and it is very fully featured. The versions for Qt,Wx and Web graphics are still in development so some testing may be required if you are hoping for full code compatibility between the different libraries.

There probably aren’t a lot of cases where you would want to flip between Qt, Wx and Tkinter graphic engines but it is remarkable that the possibility exists.

To install the default Tktinter  version of  Pysimplegui enter:

pip install pysimplegui

PySimplegui has a wide range of graphic widgets or elements. Graphic presentations are built by creating a layout variable. Graphic elements are placed in separate rows by open and closed square brackets.

gui_layout

A Button Interface Project

For my rover project I used a layout of 5 rows. The first row contains a feedback text, then rows 2-5 contains buttons.

The code below is a simple button app.


# Create a simple graphic interface
#
import PySimpleGUI as sg

layout = [ [sg.Text("the feedback" , key="feedback")],
           [sg.Button("FORWARD")],
           [sg.Button("LEFT"),sg.Button("RIGHT")],
           [sg.Button("STOP")],
           [sg.Button("QUIT")]
          ]
# Create the Window
window = sg.Window('My First App', layout)
# Event Loop to process "events"
while True:
    event, values = window.read()
    window['feedback'].Update(event) # show the button in the feedback text
    print(event,values)
    if event in (None, 'QUIT'):
        break
window.close()

The PySimplegui sg.window() call displays a window with the title and a layout definition (line 11). The window.read() will return events and values that have been changed (line 14). The feedback text element (line 5) is given a key name of  feedback, and this key name is used for updates to show the key press (line 15).

sg_basic

Standalone Web Apps with PySimpleGUIWeb

The PySimpleGUIWeb library is still under development, so be aware that not all the features in PySimpleGUI are fully supported in the Web version. PySimpleGUIWeb is an excellent way to create a lightweight standalone Web interface, but it is important to note that it isn’t designed to be a multi-page/multi-user Web environment.

To install PySimpleGUIWeb enter:

pip install remi
pip install pysimpleguiweb

The PySimpleGUIWeb window() call has a few more options, such as:

  • web_ip – the IP address to use for the PySimpleGUIWeb micro Web server
  • web_port – port on the micro Web server
  • web_start_browser – open a Web browser on app start

If you use our earlier button example but this time import PySimpleGUIWeb and add some web options we see an almost identical presentation however this time it’s in a Web interface.

sg_basic_web

Command line options can be used to toggle between the different libraries by:

import sys

# Pass any command line argument for Web use 
if len(sys.argv) > 1: # if there is use the Web Interface 
    import PySimpleGUIWeb as sg
    mode = "web"
else: # default uses the tkinter GUI
    import PySimpleGUI as sg
    mode = "tkinter"

Formatting of Display Elements

The next step is to adjust the graphic elements’  fonts, colors, and size properties.

Below is an example of changing the “FORWARD” button to have a size of 32 characters wide and 3 lines high with color and larger font.

[sg.Button("FORWARD", size=(32,3), 
  font="Ariel 32", 
  button_color=('white','green'))]

To make the interface more usable all the rover control buttons can be adjusted and the “QUIT” button can be left the default.

py_rovergui

Raspberry Pi Rover Interface

For my Raspberry Pi Rover project I used :

  • Arduino car chassis (~ $15),
  • a portable USB charger
  • Pimoroni Explorer Hat Pro (a Pi motor shield)

Below is the final code and it used a command line option (any character) to toggle into a Web application, otherwise it was the default PySimpleGUI interface. The application also included the Pi GPIO library to start/stop the car chassis motors.


# SGui_rover.py - use PySimpleGUI/Web to control a Pi Rover Pi
#

import sys
# Pass any command line argument for Web use
if len(sys.argv) > 1: # if there is use the Web Interface
    import PySimpleGUIWeb as sg
    mode = "web"
else: # default uses the tkinter GUI
    import PySimpleGUI as sg

import RPi.GPIO as gpio
gpio.setmode(gpio.BOARD)
# Define the motor pins to match your setup
motor1pin = 38 # left motor
motor2pin = 37 # right motor
gpio.setup(motor1pin, gpio.OUT)
gpio.setup(motor2pin, gpio.OUT)

# Send Action to Control Rover
def rover(action):
if action == "FORWARD":
    gpio.output(motor1pin, gpio.HIGH)
    gpio.output(motor2pin, gpio.HIGH)
if action == "LEFT":
    gpio.output(motor1pin, gpio.HIGH)
    gpio.output(motor2pin, gpio.LOW)
if action == "RIGHT":
    gpio.output(motor1pin, gpio.LOW)
    gpio.output(motor2pin, gpio.HIGH)
if action == "STOP":
    gpio.output(motor1pin, gpio.LOW)
    gpio.output(motor2pin, gpio.LOW)

# All the stuff inside your window.
myfont = "Ariel 32"
layout = [ [sg.Text(" ",size=(20,1) , key="feedback")],
[sg.Button("FORWARD", size=(32,3), font=myfont, button_color=('white','green'))],
[sg.Button("LEFT", size=(15,3), font=myfont),sg.Button("RIGHT", size=(15,3), font=myfont)],
[sg.Button("STOP", size=(32,3), font=myfont, button_color=('white','red'))],
[sg.Button("QUIT")]
]
# Create the Window
if mode == "web":
    window = sg.Window('PySimpleGUI Rover Control', layout,
        web_ip='192.168.0.106', web_port = 8888, web_start_browser=False)
else:
    window = sg.Window('PySimpleGUI Rover Control', layout )

# Event Loop to process "events" and pass them to the rover function
while True:
    event, values = window.read()
    print(event,values)
    if event in (None, 'QUIT'): # if user closes window or clicks cancel
        break
    window['feedback'].Update(event) # show the button in the feedback text
    rover(event)

window.close() # exit cleanly

py_sg_rover

Final Comment

I feel that PySimpleGUI and PySimpleGUIWeb have a lot of great potential for Raspberry Pi projects.

 

littleBit Dashboards (without Cloud Bits)

littleBits is a set of electronic components that magnetically connect together. litteBits is geared towards the kids STEP market and it is available in many schools and libraries.

The littleBits company has done an excellent job making their product easy to use. There is a large variety of different “bit” modules and for Internet applications there is a Cloud Bit ($59).

I found that the Cloud Bit was very easy to get up and running, but I found it was expensive at $59 and somewhat limiting, for example you are only access 1-input and 1-output. So if you want to do 2 inputs/output you would need to purchase a second Cloud bit module.

In this blog I’d like to document how I used a $39 Arduino Bit to do 3-inputs and 3-outputs. I also had the code talk directly to a free Web Dashboard (AdaFruit).

littleBits Arduino Program

A set of commands needs to be setup between the littlebits Arduino module and the PC/Pi. In my Arduino program I referenced the ports A,B,C as inputs (on the left side), and D,E,F as outputs (on the right side).

The commands from the PC/Pi would be : reference_pin:value, for example D:255 would set the D (top left pin) at 100%. It’s important to note that Arduino inputs and outputs are scaled from 0-255.

For inputs the littleBits would send the results as pin: reference_pin:value, for example B:255 would be the result at full scale for the A0 input.

ard_abc

My  test setup had:

  • A fork bit – this meant I only needed 1 power input source
  • A slider bit (0-1) on bit D0 (A)
  • A dimmer bit (0-255) on bit A0 (B)
  • A temperature bit on bit A1 (C)
  • An LED on bit d1 (D)
  • A number bit on D5 (E)
  • a bargraph bit on D9 (F)

lb_ard_setup

Below is the littleBits Arduino program that managed the serial communications.

// littleBits_2_Dashboards - create a serial interface to read/write to a PC/Pi
//
// Command from the littleBits: (A,B,C are the left pins) 
//  A:value <- for example B:24, pin A0 (2nd input) is 24 

// Commands from the PC/Pi: (D,E,F are the right pins)
//  D:output <- for example E:128, set pin A0 to 50% (128/255)
//
String thecmd; 
String thevalue;
String theinput;
char outstring[3];

void setup() {
  //define the littleBits right side pins 1,5 and 9 
  pinMode(1, OUTPUT);
  pinMode(5, OUTPUT);
  pinMode(9, OUTPUT);
  // define the littleBits left side inputs
  pinMode(0, INPUT);
  pinMode(A0, INPUT);
  pinMode(A1, INPUT);
  
  Serial.begin(9600); // this needs to match the PC/Pi baud rate
}
void loop() { 
  if (Serial.available() > 0) {
    thecmd = Serial.readStringUntil("\n"); 
    if (thecmd.length() > 2) { // ensure the msg size is big enough
      thevalue = thecmd.substring(2);
      if (thecmd.startsWith("D")) { analogWrite(1,thevalue.toInt()); }
      if (thecmd.startsWith("E")) { analogWrite(5,thevalue.toInt()); }
      if (thecmd.startsWith("F")) { analogWrite(9,thevalue.toInt()); }
    }     
  }
  // Try 3 different inputs: d0 = on/off , A0 = pot, A1 = temp sensor

  sprintf(outstring,"%d",digitalRead(0));
  Serial.write("A:");
  Serial.write(outstring);
  Serial.write("\n");

  sprintf(outstring,"%d",analogRead(A0));
  Serial.write("B:");
  Serial.write(outstring);
  Serial.write("\n");

// A1 is an "i12" littleBits temperature sensor
  int temp = analogRead(A1);
  temp = map(temp,0,1023,0,99); //rescale. Sensor range is 0-99 C or F
  sprintf(outstring,"%d",temp);
  Serial.write("C:");
  Serial.write(outstring);
  Serial.write("\n");
  

  delay(5000);
}

The Arduino IDE “Serial Monitor” can be used to view the output and set values.

msgbox

Python on the PC or Raspberry Pi

The Arduino program will send input data for A,B,C every 5 seconds. This input can be seen in Python by:

#
# littleBits Read Test
#
import serial

ser = serial.Serial(port='/dev/ttyACM1', baudrate=9600) # format for Linux
#ser = serial.Serial(port='COM1', baudrate=9600) # format for Windows

while True:
    inline = ser.readline()
    inline = inline.decode() # make a string
    pin = inline[0:1] # the first character is the pin
    thevalue = inline[2:-1] # the value is between ":" and "\n"<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>
    print(pin,thevalue)

The output will look something like:

A  1
B  1023
C  21

To write commands from Python:

Write
#
# littleBits Write Test
#
import serial

ser = serial.Serial(port='/dev/ttyACM2', baudrate=9600) # format for Linux
#ser = serial.Serial(port='COM1', baudrate=9600) # format for Windows

while True:
    print("\nWrite an output value to littleBit")
    out = input("Enter pin:value, pin=A,B,C example: 'E:255' : ")
    out = out.upper() + "\n"
    out2 = out.encode('utf_8')
    ser.write(out2)

Adafruit Dashboards

There are lots of good free dashboards. For this project I used the Adafruit site. To get started you will need to log in and create a free account.

I’ve bought a number of components from Adafruit. I think that they are an excellent company that goes out of their way to create great user guides and products.

To get started with Adafruit Dashboards see: https://github.com/adafruit/Adafruit_IO_Python

The first step is to add some Adafruit tags that the code can read/write to.

Ada_feeds

In the Python code a couple of dictionaries (lb_inputs, lb_outputs)  were created to link the littleBit references (A-F) with the Adafruit tags. Also two dictionaries (lb_inlast, lb_outlast) are used to minimize communications traffic so that only new values were written.

#
# Import standard python modules
import time, random
import serial

# import Adafruit IO REST client
from Adafruit_IO import Client, Feed, RequestError

ADAFRUIT_IO_USERNAME = "put_your_username_here"
ADAFRUIT_IO_KEY = "c039f24ecb6...xxxxx"

aio = Client(ADAFRUIT_IO_USERNAME, ADAFRUIT_IO_KEY)

# Create dictionaries of inputs, output, and last values
lb_inputs = {"A":"lb-slide", "B":"lb-dimmer","C": "lb-temp"}
lb_inlast = {"A":0, "B":0,"C": 0}
lb_outputs = {"D":"lb-led", "E":"lb-number", "F":"lb-bar"}
lb_outlast = {"D":0, "E":0,"F": 0}

# Setup the serial port connection
ser = serial.Serial(port='/dev/ttyACM1', baudrate=9600)

while True:
    # Get values from littleBits and write to the dashboard
    inline = ser.readline()
    inline = inline.decode() #inline should look like: A:125\n
    pin = inline[0:1] # pin is the first character in string
    thevalue = inline[2:-1] # value is between ":" and "\n"
    if lb_inlast[pin] != thevalue: # Write only new values
        print(pin,thevalue, lb_inputs[pin])
        ada_item = aio.feeds(lb_inputs[pin])
        aio.send(ada_item.key,thevalue)
        lb_inlast[pin] = thevalue

    thetag = 'lb-slide'
    # Write new dash values to littleBits if they've changed
    for lbtag, dashtag in lb_outputs.items():
        print (lbtag,dashtag)
        thevalue = aio.receive(dashtag).value
        if lb_outlast[lbtag] != thevalue: # Write only new values
            outstr = lbtag + ":" + thevalue + "\n"
            print(outstr)
            ser.write(outstr.encode('utf_8'))
            lb_outlast[lbtag] = thevalue   

    time.sleep(2)

If everything is working correctly then new values should be written to in both directions. On the Adafruit Web site the Feed page should show the new values.

To make things look more presentable Adafruit Dashboards can be used.

ada_dash

Final Comments

In this project I used the Adafruit API, other good platforms would be IFTTT and Node-Red

Arduino Yún for IoT

There are some great Internet of Things hardware platforms, such as Raspberry Pi’s, ESP-32 based Arduino modules, Particle and Beaglebone to name just a few. Each of these systems have their strengths and weakness. Some are strong on the hardware side, like the Arduino modules, and others excel on the programming side like the Raspberry Pi.  

The Yún is somewhat unique in that it’s a module with two processors, one that supports standard Arduino programming and a second processor that supports Linux and the OpenWrt wireless stack. The Yún ($59) has an Arduino Uno form factor and there are clones like the LinkIt Smart 7688 Duo ($18) in an Arduino Nano form factor.

yun_overview

In this blog I wanted to document some of key feature and functions that I worked through, namely:

  • moving file – scp and ftp
  • Python bridging to Arduino
  • uhttpd Web Server – with Python CGI
  • Yún REST API
  • MQTT
  • Yún mailbox

Yún Overview

The Yún, now in revision 2, seems to have been somewhat overlooked because of the all the low cost ESP-8266 and ESP32 based Arduino modules starting around $2.

Some the interesting features of the Yún include:

  • Arduino code can launch and get feedback from Linux apps
  • A bridge library allows Linux programs and Arduino to share data
  • Arduino libraries for:
    • Web client
    • Web server
    • Mailbox to Linux
  • Yún has a lightweight web server, uhttpd, that can be used to Python, PHP, Lua CGI web programs
  • Yún has a read/write REST API web service

Getting Started

The Arduino side of the Yún is available “out of the box” like any other Arduino module when you connect a USB cable into the module.

The Linux side however requires some configuration. Please see one of the guides on the Linux setup. Once your module is connected to your network, the Yun webserver can be used to configure features and add software components.

yun_software1

Another option for loading software is to use the opkg package manager from an SSH connection.

yun_opkg

So for example to install nano, the simple text editor, enter:

 opkg install nano

Moving File and Working in OpenWrt Linux

The OpenWrt Linux isn’t loaded with with an X-windows environment and this means that you can not run idle or leafpad etc. to do easy editing of program files. Nano is a good command line text editor but it’s not the same as a windows based editor.

To move files between a PC and OpenWrt you have some options. The two that I like are:

  • scp – secure copy, this is built in both OpenWrt and Microsoft Window 10
  • ftp – file transfer protocol. Needs to be installed in OpenWrt.

There are a few ftp servers that could be installed in OpenWrt, vsftp is a lightweight option and it can be installed by:

opkg update
opkg install vsftpd

Once vsftpd is installed it needs to be enabled, this can be done from the command line or via the web interface.

yun_ftp_start.png

Yún Bridge

The Yun Arduino bridge library allow variables to be passed between the Arduino code and a Python program. Below is an example that writes two random variables available, and it creates a “bridge1” variable that can be written from the Python side.

// Simple Yun Bridge Example
//

#include <Bridge.h>
// create a bridge variable to get remote data
char bridge_Value[10];

void setup()
{
    Bridge.begin();     // this launches /usr/bin/run-bride on Linino
}

void loop()
{    
    // create 2 random bridge values, that are sourced from the Arduino
    Bridge.put("random1", String(random(1, 100)));
    Bridge.put("random2", String(random(1, 100)));
    // Called the bridge value "bridge1". This is the name used on the Python side
    Bridge.get("bridge1", bridge_Value, 6);
    delay(1000);  
}

An example to read/write in Python:


#!/usr/bin/python

import sys

sys.path.insert(0, '/usr/lib/python2.7/bridge/')

from bridgeclient import BridgeClient as bridgeclient
value = bridgeclient()

message = value.get("random1") # get a value from Arduino

print "Random1: ", message

value.put("bridge1","1111")  # set a value to Arduino

Yún uhttpd web server

The uhttp web server is used for Yun setup and configuration. This web server can also be used to for custom static pages and user web apps. To view/modify the web server settings:

nano /etc/config/uhttpd

Within this config file, you can enable custom web applications by defining an interpreter:

# List of extension->interpreter mappings.
# Files with an associated interpreter can
# be called outside of the CGI prefix and do
# not need to be executable.
# list interpreter ".php=/usr/bin/php-cgi"
# list interpreter ".cgi=/usr/bin/perl"
list interpreter ".py=/usr/bin/python

# Lua url prefix and handler script.
# Lua support is disabled if no prefix given.
# option lua_prefix /luci
# option lua_handler /usr/lib/lua/luci/sgi/uhttpd.lua

The default directory for user programs is: /www/cgi-bin

Python CGI – Get Values

To read the Arduino bridge values in a Python CGI program, add a file to the /www/cgi-bin directory. For my example I called the file p1.py :


#!/usr/bin/python

import sys
import cgi

sys.path.insert(0, '/usr/lib/python2.7/bridge/')

from bridgeclient import BridgeClient as bridgeclient
value = bridgeclient()

print "Content-type:text/html\r\n\r\n"
print '<html>'
print '<head>'
print '<title>Python CGI Bridge</title>'
print '</head>'
print '<body>'
print '<h2>Python CGI Bridge</h2>'
print 'Random Value1: ' + value.get("random1")
print 'Random Value2: ' + value.get("random2")
print '</body>'
print '</html>'

To run the python script on the web page you will need to change the file right to executable:

chmod +x p1.py

You can debug and see output from the command line.


root@yun1:/www/cgi-bin# ./p1.py
Content-type:text/html

<html>
<head>
<title>Python CGI Bridge</title>
</head>
<body>
<h2>Python CGI Bridge</h2>
Random Value1: 13
Random Value2: 24
</body>
</html>

If the output looks good, try the app from the Web page:

CGI_bridge1

Python CGI – Put Values

There are a number of methods that can be used to send user input from a web page. A simple approach is to use a form. The form data can be read from the cgi.FieldStorage object, using a form.getvalue() call.

#!/usr/bin/python

# Import modules for CGI handling
import cgi
import sys

sys.path.insert(0, '/usr/lib/python2.7/bridge/')
from bridgeclient import BridgeClient as bridgeclient
value = bridgeclient()

# Create instance of FieldStorage
form = cgi.FieldStorage()

# Get data from fields
bridge1 = form.getvalue('bridge1')
if bridge1 != "" :
    value.put("bridge1",bridge1)


print "Content-type:text/html\r\n\r\n"
print """
<html>
<head>
<title>Python CGI - Put Bridge Value</title>
</head>
<body>
<h2>Python CGI Form - Put Bridge Value</h2>
<form action = "/cgi-bin/p2.py" method = "post">
Enter BRIDGE1 value: <input type = "text" name = "bridge1"><br />

<input type = "submit" value = "Submit" />
</form>
</body>
</html>"""

 

The web page will call itself when the submit button is pressed.

CGI_bridge2

Yún REST API

The Yún REST API is a web service that allow remote users and web browsers to view and set bridge values. I found the REST API to be a good tool for testing my Web CGI and Python applications

To view all bridge variables enter: http://yourYun_IP/data/get

yun_rest_1

To get a specific bridge value enter:  http://yourYun_IP/data/get/my_bridge_value

yun_rest_2

To put a bridge value enter: http://yourYun_IP/data/put/my_bridge_value/value 

yun_rest_3

IoT Connections – MQTT

For Internet of Things (IoT) projects you need to pass data from the Arduino and some server. The communications and the server would be something like MQTT or Redis.

The Yún does not have direct access to the Wifi or ethernet port so the standard Arduino libraries for MQTT or Redis etc. will not work. An alternative approach is load the protocol’s command line client on the Linux side and then Arduino can shell out to the Linux tool.

For example to load the  Mosquitto MQTT command line tools:

opkg update
opkg install mosquitto-client

To test MQTT publishing to a topic (mytag1) with a message of 249 on remote client (192.168.0.116) :

mosquitto_pub -h 192.168.0.116 -u username -P password -t mytag1 -m 249

To remotely subscribe to

mosquitto_sub -h 192.168.0.116 -u username -P password -t mytag1

An example of sending out four MQTT messages in Arduino:

/*
Shell out to pass values to a MQTT command line pub
*/
#include <Process.h>

void setup() {
  Bridge.begin();	// Initialize the Bridge
  Serial.begin(9600);	// Initialize the Serial

  // Wait until a Serial Monitor is connected.
  while (!SerialUSB);
}

void loop() {
  Process p;
  String thecmd;
  String strval = String( random(0,100));
  // create a string with host, username and password, -t is for the topic
  String theparams = "-h 192.168.0.116 -u pete -P pete -t ";
  int numtopics = 4;
  String topics[4] = {"tag1","tag2","tag3","tag4"};

  for (int i=0; i < numtopics ; i++) {
    strval = String( (i*100) + random(1,99)); // create a random value - 0,100+,200+,300+
    thecmd = "mosquitto_pub " + theparams + topics[i] + " -m " + strval;
    Serial.println(thecmd);
    p.runShellCommand(thecmd);
    // do nothing until the process finishes, so you get the whole output:
    while (p.running());
  }
  // Wait 5 seconds and repeat
  delay(5000);  // wait 5 seconds before you do it again
}

It is also possible to create some bridge variables and pass them to a Python program that could do the MQTT communications.

Yún Mailbox

At present the mailbox only works in one direction, and this into the Arduino. Both the REST web interface and Python have writing functionality but no read capabilities (on the forums this has been identified, so a future revision may add this).

To write using REST:

http://yun_ip/mailbox/mymessagetext

To write using Python:


import sys
sys.path.insert(0, '/usr/lib/python2.7/bridge')
from bridgeclient import BridgeClient
client = BridgeClient()
client.mailbox("my_message")

The Arduino code to read message:

// Mailbox Read Example
//
#include <Mailbox.h>

void setup() {

  // Initialize Bridge and Mailbox
  Bridge.begin();
  Mailbox.begin();

  // Initialize Serial
  SerialUSB.begin(9600);

  // Wait until a Serial Monitor is connected.
  while (!SerialUSB);
  SerialUSB.println("Mailbox Read Message\n");
}

void loop() {
  String message;

  // if there is a message in the Mailbox
  if (Mailbox.messageAvailable()) {
    // read all the messages present in the queue
    while (Mailbox.messageAvailable()) {
      Mailbox.readMessage(message);
      SerialUSB.println(message);
    }

    SerialUSB.println("Waiting 10 seconds before checking the Mailbox again");
  }
  // wait 30 seconds
  delay(10000);
}

I’m not totally sure when I’d use a mailbox. The mailbox is a generic message and it’s not queued (only 1 message) so I think that using the standard bridging of values with get/put is more useful.

Final Comments

The Yun is missing a lot of networking libraries that are available with the ESP-8266 and ESP32 family of Arduino modules. In some cases like with MQTT that was a good Linux command line tool that could be used, but in cases like where you want to connect to a Redis server, you might have some challenges.

I really liked Yun’s built in uhttpd web server, this is far superior to the ESP-8266 Arduino web server library calls.

I found that putting logic into a combination of Python and Arduino could be a little confusing, but for projects with a lot of text or json data using Python would be a real plus. Also for projects where multitasking is required use the Linux process would be ideal.

RabbitMQ REST API

RabbitMQ is an open source middleware message solution that natively uses AMQP communications but it has a good selection of plug-ins to support features like: MQTT, MQTT Web Sockets, HTTP REST API and server-to-server communications.

rabbitmq_overview

RabbitMQ has a very good selection of interfacing clients for most of the major programming languages. For applications where there is no custom programming library the REST API can be used.

Applications where the REST API could be used might include:

  • Arduino IoT projects
  • Web Clients (not using NodeJS)
    • For simple reading and writing of messages
    • For monitoring of RabbitMQ loading
  • AppInventor Android apps

Unfortunately I didn’t find the RabbitMQ REST API to be 100% straightforward, so in this blog I would like to document some of my notes and testing.

What is REST API

Representational State Transfer (REST) is a software style for creating Web services. A client issues an HTTP request to a server with a GET/POST/PUT or DELETE method and the server responds with a JSON (Javascript Object Notation) formatted message.

RESTAPI_overview

The most common tool to test REST API is curl. Curl is supported on all major OS’s. The curl syntax will vary slightly between Windows and Linux, especially in how the systems handle strings.

RabbitMQ Setup

RabbitMQ can be installed on Window, Linux, MacOS systems and there are also some cloud based offerings. For small systems lower end hardware like a Raspberry Pi can be used.  For complete RabbitMQ installation instructions see: https://www.rabbitmq.com/download.html . To install and run RabbitMQ on a Ubuntu system enter:

sudo apt-get update
sudo apt-get install rabbitmq-server
sudo service rabbitmq-server start

The next step is to add some plug-ins. For the Web Administration and REST API plug-ins:

sudo rabbitmq-plugins enable rabbitmq-management

The rabbitmqctl command line tool allows you to configure and review the RabbitMQ server. To add a user pete,  with password pete, that has config, write and read rights for management and administrator access, enter:

sudo rabbitmqctl add_user pete pete
sudo rabbitmqctl set_permissions -p / pete ".*" ".*" ".*"
sudo rabbitmqctl set_user_tags pete management administrator

After you’ve defined an administrative user the RabbitMQ web management plug-in can be accessed by: http://ip_address:15672 .

Rabbit_webadmin

Once the Web Management component is installed it is much easier to test and play with REST API calls.

REST API Calls Using CURL

For a complete set of RabbitMQ REST API calls see: https://pulse.mozilla.org/api/ .

Typically you would use the RabbitMQ tools to create and exchanges and queues but it is possible to do this via the REST API interface.

Below are some examples using user: pete and a password of pete . Note: %2f is used for default (blank name) virtual host.

Create an exchange: webex

curl -i -u pete:pete -H “content-type:application/json” -X PUT http://192.168.0.116:15672/api/exchanges/%2f/webex -d”{“”type””:””direct””,””auto_delete””:false,””durable””:true,””internal””:false,””arguments””:{}}”

Create a Queue: webq1

curl -i -u pete:pete -H “content-type:application/json” -X PUT http://192.168.0.116:15672/api/queues/%2f/webq1 -d”{“”auto_delete””:false,””durable””:true,””arguments””:{}}”

Create Binding: Bind queue webq1 to exchange webex with a routing key of webq1

curl -i -u pete:pete -H “content-type:application/json” -X POST http://192.168.0.116:15672/api/bindings/%2f/e/webex/q/webq1 -d”{“”routing_key””:””webq1“”,””arguments””:{}}”

Comments: you need to bind a queue to an exchange if you wish to write to it. The routing key does the binding. The key can be any name but using the queue name makes it simple. A queue can also have binding to multiple exchanges.

Get a message: from queue webq1

curl -i -u pete:pete -H “content-type:application/json” -X POST http://192.168.0.116:15672/api/queues/%2f/webq1/get -d”{“”count””:5,””requeue””:true,””encoding””:””auto””,””truncate””:50000}”

Comments: A queue can have multiple messages. A requeue=false, will remove messages that are already read.

Post a message (44.4) : to exchange webex with routing key webq1

curl -i -u pete:pete -H “content-type:application/json” -X POST http://192.168.0.116:15672/api/exchanges/%2f/webex/publish -d”{“”properties””:{},””routing_key””:””webq1“”,””payload””:””44.4“”,””payload_encoding””:””string””}”

 

Javascript REST API Client

I had some Web browser issues when my test PC and my RabbitMQ server were in different domains. Web Browsers like Chrome and  Firefox will complain about a CORS policy problem.

Cross-origin resource sharing (CORS) is a mechanism that allows restricted resources on a web page to be requested from another domain outside the domain from which the first resource was served.

If you are crossing domains and you want to do some quick testing Microsoft Explorer will work.

Javascript GET a Queue Message

I used the AJAX XMLHttpRequest object to do my GET and POST requests.

<!DOCTYPE html>
<html>
<body>
<h2>RabbitMQ REST API - Get Queue Message</h2>

URL/API Call: <input type="text" id="theurl" value="http://192.168.0.116:15672/api/queues/%2f/"  size = "50" /> <br><br>
Queue Name:   <input type="text" id="qname" value="webex"/> <br><br>
Requeue: <input type="text" id="rq" value="true" /> (false = remove old values)<br><br>
Message Count: <input type="text" id="mcount" value="5"/> <br><br>
<button type="button" onclick="sendoutput()">Get Message(s)</button>
<br>

<hr>
<p id="demo"></p>

<script>
function sendoutput() {
	// RabbitMQ username and password
	var username = "pete";
	var password = "pete";
	var xhttp = new XMLHttpRequest();
	
    var fullurl = document.getElementById("theurl").value + document.getElementById("qname").value + "//get";
	
	xhttp.open("POST", fullurl, true,username,password);
	
	// change to your Auth username and password

	xhttp.setRequestHeader("Content-Type","application/jsonp");

	var rq = document.getElementById("rq").value;
	var mcount = document.getElementById("mcount").value;
	var params = JSON.stringify({"count":mcount,"requeue":rq,"encoding":"auto","truncate":"50000"});
	// Define a handler function when the response comes back
	xhttp.onreadystatechange = function() {
		// Get the response and put on it the screen
		if (this.readyState == 4 ) {	
			document.getElementById("demo").innerHTML = "Result=" +xhttp.responseText;
		}
	}

	xhttp.send(params);
}

</script>
</body>
</html>

r_get

Javascript POST an Exchange Message

Unfortunately you can not send a message directly to a queue, instead you need to send a message to an exchange with routing key. This routing key is linked to the queue that you wish to connect to.

<!DOCTYPE html>
<html>
<body>
<h2>RabbitMQ REST API - An Exchange Message</h2>

URL/API Call: <input type="text" id="theurl" value="http://192.168.0.116:15672/api/exchanges/%2f/"  size = "50" /> <br><br>
Exchange Name:   <input type="text" id="exname" value="webex"/> <br><br>
Routing Key:   <input type="text" id="rkey" value="webq1"/> <br><br>
Value to Send:   <input type="text" id="theval" value="11.1"/> <br><br>

<button type="button" onclick="sendoutput()">Send Output</button>
<br>

<hr>
<p id="demo"></p>

<script>
function sendoutput() {
	// RabbitMQ username and password
	var username = "pete"; 
	var password = "pete";  
	
	var xhttp = new XMLHttpRequest();
	
	var fullurl = document.getElementById("theurl").value + document.getElementById("exname").value + "//publish";
	
	xhttp.open("POST", fullurl, true,username,password);

	xhttp.setRequestHeader("Content-Type","application/jsonp");
	
	var rkey = document.getElementById("r").value
	var theval = document.getElementById("theval").value
	var params = JSON.stringify({"properties":{},"routing_key":rkey,"payload":theval,"payload_encoding":"string"});
	
	// Define a handler function when the response comes back
	xhttp.onreadystatechange = function() {
		// Get the response and put on it the screen
		if (this.readyState == 4 ) {	
			document.getElementById("demo").innerHTML = "Result=" +xhttp.responseText;
		}
	}
	xhttp.send(params);
}
</script>
</body>
</html>

r_post

Python – get message(s) from a queue

The requests library is extremely simple to use for REST interfacing. It is installed by:

pip install requests

# importing the requests library
import requests

# defining the api-endpoint
API_ENDPOINT = "http://192.168.0.116:15672/api/queues/%2f/webq1/get"

headers = {'content-type': 'application/json'}
# data to be sent to api
pdata = {'count':'5','requeue':'true','encoding':'auto','truncate':'50000'}

# sending post request and saving response as response object
r = requests.post(url = API_ENDPOINT ,auth=('pete', 'pete'), json = pdata, headers=headers)

# extracting response text
pastebin_url = r.text
print("Response :%s"%pastebin_url)

Python – Write a message to an exchange


# importing the requests library
import requests
import json

# defining the api-endpoint
API_ENDPOINT = "http://192.168.0.116:15672/api/exchanges/%2f/webex/publish"

# your source code here

headers = {'content-type': 'application/json'}
# data to be sent to api

pdata = {'properties':{},'routing_key':'webq1','payload':'90.4','payload_encoding':'string'}

# sending post request and saving response as response object
r = requests.post(url = API_ENDPOINT ,auth=('pete', 'pete'), json = pdata, headers=headers)

# extracting response text
pastebin_url = r.text
print("Response :%s"%pastebin_url)

Final Comments

The RabbitMQ REST interface is not designed for high throughput, but it is an excellent way to connect devices that have no native API support.

For my next round of testing I’d like to work on an Arduino and a AppInventor Android app.