Home Assistant History on Node-Red Charts

April 7, 2021Posted in IoT, Node-RedTagged HA, History, Home Assistant, Node-Red, NodeRed, Trend1 Comment

Home Assistant is an open source home automation platform that can monitor and control smart home devices and it integrates with many of other common systems.

Home Assistant installation is targeted for Raspberry Pi’s but other hardware options are available.

I was very impressed how easy it was to install Home Assistant and get a basic home integration system up and running.

There is a huge number of integration solutions (1500+) that connect to most of the mainstream products. However if you want to do some custom programming with connections to Arduino or other Raspberry Pi or PCs there isn’t an easy “out of the box” solution. To solve this requirement Home Assistant has included Node-Red as an add-on.

Node-RED is a visual programming tool for wiring together hardware devices, APIs and online services.

For information how to install Node-Red on Home Assistant see the HA documentation. (I wrote a blog for my installation).

Home Assistant History

The default installation of Home Assistant has history enabled, and the data is stored in local SQLite database (home-assistant_v2.db) within your configuration directory unless the recorder integration is set up differently.

Charts of sensor history can be show in the Home Assistant Overview pages or as dialogs.

Using Node-Red with HA History

The Node-Red installation has a number of Home Assistant nodes that allow sensors data to be read and created. For viewing history the HA get history node is used.

A simple manual test circuit to get history would have: an injector, a get_history and a debug node.

By double-clicking on the get history node its configuration can be defined. For this example a sensor entity id of sensor.cpu_temp is used with a 10 minute (10m) time scale. When the injector is toggled the debug window will show an array to data and time entries.

3 Button History Chart

The logic to create a 3 button history chart would use: 3 dashboard buttons, 3 get history nodes, a Javascript function, and a dashboard chart node.

The Node Red chart node typically does local historical storage within the node itself. However the chart node can also be used to read external data and show a line chart of the data,nothing is stored locally (more info on this).

A javascript function node can be used to format the data into the required form:

//
// Format the HA History results to match the charts JSON format
//

var series = ["HA Values"];
var labels = ["Data Values"];
var data = "[[";
var thetime;
 
for (var i=0; i < msg.payload.length; i++) {
    thetime = (msg.payload[i].last_changed); // Note: check your format?
    data += '{ "x": "' + thetime + '", "y":' + msg.payload[i].state + '}';
    if (i < (msg.payload.length - 1)) {
        data += ","
    } else {
        data += "]]"
    }
}
var jsondata = JSON.parse(data);
msg.payload = [{"series": series, "data": jsondata, "labels": labels}];
return msg;

The payload data will be in the format of:

If everything is formatted correctly the Node Red dashboard page should look like:

For a more flexible presentation it would be good to use adjustable time periods.

Final Thoughts

For simple historical storage I found that the built-in History worked fine, however if you’re looking for custom long term storage then the InfluxDB add-on to HA might be a better solution.

MySQL with Arduino and Node-Red

April 25, 2020April 29, 2020Posted in Arduino, Node-RedTagged Arduino, Mysql, NodeRed, sql1 Comment

In this blog I would like to document a project that I did with my kids. The goal of the project was to learn SQL in a couple of different environments.

We did most of our initial testing/learning directly on the MySQL server. Once we got everything worked out we moved to Arduino, then Node-Red.

For our MySQL testing we used alwaysdata.com which has a free low use account. They run MariaDB which looks exactly like MySQL.

MySQL Setup

The plan was to have the Arduino module write 3 values:

randint – a random integer from 1-100
saw – a saw tooth integer from 0-100
sstatus – a text value of : HIGH, OKAY or LOW

We didn’t want to generate a time stamp from Arduino, instead we wanted MySQL to generate it. Our SQL structure was:

The field “thetime” was defined as a timestamp with CURRENT_TIMESTAMP as the default. By doing this we only needed to INSERT the Arduino data and MySQL would add the current timestamp.

After the Arduino Tables was created and tested we were able to start inserting data from Arduino.

Arduino and MySQL

There is an excellent MySQL library for Arduino, that can be added via the Arduino IDE “Manage Libraries” option.

Our Arduino code wrote the 3 values to MySQL every 10 seconds.

/*
  MySQL Connector/Arduino Example : connect by wifi
*/
#include <ESP8266WiFi.h>           // Use this for WiFi instead of Ethernet.h
#include <MySQL_Connection.h>
#include <MySQL_Cursor.h>

IPAddress server_addr(111,222,111,111);  // IP of the MySQL *server* here
char user[] = "yourusername";              // MySQL user login username
char password[] = "yourpassword";        // MySQL user login password

// WiFi card example
char ssid[] = "xxxxx";         // your SSID
char pass[] = "xxxxx";     // your SSID Password

WiFiClient client;                 // Use this for WiFi instead of EthernetClient
MySQL_Connection conn(&client);
MySQL_Cursor* cursor;

int saw = 0;

void setup()
{
  Serial.begin(9600);

  // Begin WiFi section
  Serial.printf("\nConnecting to %s", ssid);
  WiFi.begin(ssid, pass);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  // print out info about the connection:
  Serial.println("\nConnected to network");
  Serial.print("My IP address is: ");
  Serial.println(WiFi.localIP());

  Serial.print("Connecting to SQL...  ");
  if (conn.connect(server_addr, 3306, user, password))
    Serial.println("OK.");
  else
    Serial.println("FAILED.");
  
  // create MySQL cursor object
  cursor = new MySQL_Cursor(&conn);
}

void loop()
{
  // Sample insert query
  //
  char INSERT_SQL[] = "INSERT INTO datal (randint, saw, sstatus) VALUES ('%d','%d','%s')";
  char query[128];
  int randint = random(100);


saw++;
if (saw == 100){
  saw = 0;
}

  if (randint <= 10){
     sprintf(query, INSERT_SQL, randint, saw, "LOW");
  } else if (randint >= 90){
     sprintf(query, INSERT_SQL, randint, saw, "HI");
  } else {
     sprintf(query, INSERT_SQL, randint, saw, "OKAY");
  }
  
  Serial.println(query);

  if (conn.connected())
    cursor->execute(query);


  // Sample select query - Get the last 2 values and show them 
  // Table: arduino1 - this is written to from Node-Red 

  int arddata;
  // Initiate the query class instance
  row_values *row = NULL;
  MySQL_Cursor *cur_mem = new MySQL_Cursor(&conn);

  char squery[] = "SELECT thetime,arddata FROM arduino1 ORDER BY thetime DESC limit 2;";
  cur_mem->execute(squery);
  // Fetch the columns (required) but we don't use them.
  column_names *columns = cur_mem->get_columns();

  do {
    row = cur_mem->get_next_row();
    if (row != NULL) {
      //thetime = (row->values[0]);
      arddata = atoi(row->values[1]);
      Serial.print("Time: "); Serial.print(row->values[0]); Serial.print(" data: "); Serial.println(arddata);
      
    }
  } while (row != NULL);
  delete cur_mem;
  
  delay(10000);
  
}

MySQL Views

The table(s) that we created were nice and simple for Arduino data, but they couldn’t be used directly for any types of statistics like daily/hourly/minute averages/maxs or minimums. So we created some MySQL views.

The first things that we needed to do was create some “time” columns like HOUR and MINUTE.

CREATE VIEW v_raw as 
 SELECT HOUR(thetime) as HOUR, 
   MINUTE(thetime) as MINUTE, 
   randint, saw, sstatus from datal 
   order by thetime DESC

This SQL statement creates a view (v_raw) that shows the hour, minute fields with data ordered from newest to oldest. The query of : SELECT * FROM v_raw; gives:

We then created a number of statistic views like:

CREATE VIEW v_min AS 
  SELECT HOUR(thetime) as HOUR, MINUTE(thetime) as MINUTE, 
    AVG(randint), AVG(saw) 
   from datal 
     GROUP BY HOUR,MINUTE ORDER BY thetime LIMIT 720

This gave us some minute averages:

Node-Red with MySQL

The component node-red-node-mysql can be added manually or via the “Manage Palettes” option.

For our project we wanted to show:

Last 100 Arduino values
Latest Average Minute Values
Latest Average Hourly Value

Then we wanted to Insert a new 0/1 value that could be read back in the Arduino. This inserted value could be used to turn on a relay connected to the the Arduino (like a start a fan, turn on a light etc.)

The logic used Dashboard buttons (to select an action). MySQL nodes do the SELECT or INSERT command, and a Dashboard Table to show the results.

The buttons are configured to pass the SQL views statement to the MySQL nodes, with the results showing in a Dashboard table node.

After the logic is complete, use the Deploy button to make the logic live. The web dashboards can be viewed at: http://yourNodeRed_IP:1880/ui . Below is an example of 1) Minute Averages, and 2) Last 100 (raw values).

Final Comments

For this project we kept things simple, and used tables to show the results. As a next step it would be good to show the results in a trend chart. In a previous blog I looked at putting SQLite into a Node-Red chart. I would use this technique with MySQL data.

Micro:bits and Node-Red

February 3, 2020June 29, 2023Posted in Node-Red, Raspberry PiTagged dashboard, IoT, micro:bit, Node-Red, NodeRed2 Comments

BBC Micro Bit, (micro:bit) is an open source hardware ARM-based embedded system designed by the BBC for use in computer education in the UK. The device is half the size of a credit card and has an ARM Cortex-M0 processor, accelerometer and magnetometer sensors, Bluetooth and USB connectivity, a display consisting of 25 LEDs, and two programmable button.

Depending on where you purchase it the price ranges between $15-$20. So this is a very attractive module for the beginning programmer.

The micro:bit module has 2 buttons to interface to it and a small 5×5 LED screen. This is good for small tests but its a little limiting.

For the most part micro:bit is a standalone unit so in this blog I wanted to show how to put micro:bits information on to a Node-Red web dashboard that could be viewed from a smart phone, tablet or PC.

Micro:bits Setup

The micro:bits has a USB connection that can be used for communications to PCs or Raspberry Pi’s. For my setup I used a Raspberry Pi Zero W, with a microUSB-to-USB adapter to connect into the micro:bit.

The micro:bit can be programmed via a nice Web Interface, for details see: https://microbit.org/guide/quick/. For this application I programmed with blocks.

My logic had the temperature and light sensor values written out ever 10 seconds, in the format of: T=xxx, L=xxx, I used a comma separator between the data pieces. Button presses were sent as either A=1, or B=1, .

Node-Red Setup

Node-Red is pre-install on the Raspberry Pi image, if you want to use a PC instead see the Node-Red installation documentation.

A Node-Red has a Serial port component (https://flows.nodered.org/node/node-red-node-serialport) that can be loaded manually or via the Palette Manager.

The first step is to insert a serial input node and define the serial interface. Double-click on the serial input node and edit the serial connection. The interface will vary with your setup but node-red will show a list of possible USB ports. The default baud rate of the micro:bits USB port is 115200. I used a timeout of 200ms to get the messages, but you could also look for a terminating character (the comma “,” could be used).

The logic used 4 Javascript function nodes to parse the micro:bit message

“Get Temp Value” Function:


// Pull out the temperature
//
var themsg = msg.payload;

if (themsg.indexOf("T=") &gt; -1) {

var msgitems = themsg.split(",");

var temp = msgitems[0];
temp = temp.substring(2,4)
msg.payload = temp;
return msg;
}


“Get Light Value” Function:
// Pull out the Light Sensor Value
//
var themsg = msg.payload;

if (themsg.indexOf("T=") > -1) {

var msgitems = themsg.split(",");

var light = msgitems[1];

light = light.substring(2,5)

msg.payload = light;

return msg;

}


“Check Button A” Function:
// If the message is Button A pressed
// "A=1,"
if (msg.payload == "A=1,") {
msg.payload =  1;
return msg;
}

“Check Button B” Function:
// If the message is Button B pressed
// "B=1,"
if (msg.payload == "B=1,") {
msg.payload =  1;
return msg;
}

Chart nodes are used to show the results. (Note: you’ll need to create a dashboard name).
For the button presses a 1-0 transition is needed after a button press, otherwise the chart will always show a value of 1. The 0-1 transition is done using a trigger node.
The final web dashboard is available at: http://your_node_red_ip:1880/UI.

Final Comments
The next step will be to add the ability to have Node-Red write values to the micro:bit. This would be done with the Node-Red serial output node. Micro:bit’s have a serial read function that would then process the command.

InfluxDB with Node-Red

February 1, 2020April 7, 2021Posted in Node-Red, Raspberry PiTagged Influxdb, IoT, Node-Red, NodeRed26 Comments

There are a lot of excellent databases out there. Almost all databases can support time tagged information and if you have regularly sampled data everything works well. However if you have irregularly sampled data things can get a little more challenging.

InfluxDB is an open-source time series database (TSDB). It is written in Go and optimized for fast, high-availability storage and retrieval of time series data in fields such as operations monitoring, application metrics, Internet of Things sensor data, and real-time analytics.

InfluxDB has a number of great features:

when data is added, a time stamp is automatically added if it’s already incluced.
InfluxDB manages aggregation of times (i.e. means over the hour)
Open Source Web Trending packages like Grafana and Chronograf will talk directly to InfluxDB
an SQL language with a time based syntax

In this blog I wanted to document my notes on:

How to add sampled data from Node-Red to Influx
How to view Influx historical data in a Node-Red chart

Why Use Node-Red with Influx

With the great Web trending interfaces like Grafana and Chronograf why use Node-Red?

I really like Grafana, but I didn’t find it to be 100% mobile friendly, whereas Node-Red is designed for mobile use.
if you’re inputting data or doing logic in Node-Red it makes sense to keep the interface logic there also.

The downside of using Node-Red is that you will have to make your own charting controls.

Getting Started with InfluxDB

The official installation document lists the various options based on your OS. For a simple Raspberry Pi or Ubuntu installation I used:

sudo apt-get install influxdb

The influxdb configuration/setup is modified by:

sudo nano /etc/influxdb/influxdb.conf

After configuration changes Influx can be restarted by:

sudo service influx restart

The Influx command line interface (CLI) is useful for getting started and checking queries. It is started by entering: influx (Note: it might be slow to initially come up).

Below I’ve opened the influx CLI and created a new database called nrdb.

~$ influx
Connected to http://localhost:8086 version 1.7.9
InfluxDB shell version: 1.7.9
> create database nrdb
> show databases
name: databases
name
----
_internal
pidata
nrdb
>

Node-Red and Influx

Node-Red is pre-installed on Raspberry Pi. If you need to install Node-Red on a Window, MacOS or Linux node see the installation instructions.

For my testing I used the following definitions:

nrdb – the InfluxDB database
mytemps – the measurement variable for my temperatures
Burlington, Hamilton – two locations for the temperatures
temperatures – the actual temperatures

Two Node-Red libraries were installed:

BigTimer – https://flows.nodered.org/node/node-red-contrib-bigtimer
InfluxDB – https://flows.nodered.org/node/node-red-contrib-influxdb

These libraries can either be installed using npm or within Node-Red using the “Manage Pallet” option.

For this project I create two sets of logic. The first set used the BigTimer to write a new simulated input every minute (via the middle output pin of BigTimer), or manual push in a value. The second part of the logic used a selected time to query the data and present it to a chart and table.

The first step is to drop a InfluxDB outpt and then configure the Influx server, table and measurements.

A Javascript function node (“Simulate an Input”) is used to format the fields and values. The first passed item is the key item, and the second parameter is a tagged value. Note: there are a number of different ways to use this node.

The Big Timer middle output will send a value out every minute. I added an Inject Node (“Force Test”) so I could see more values.

To test that things are running, the influx cli can be used:

> use nrdb
Using database nrdb
> show measurements
name: measurements
name
----
mytemps

> select * from mytemps
name: mytemps
time location temperature
---- -------- -----------
1580584703785817412 Burlington 17
1580584706364427345 Burlington 5
1580584761862704310 Burlington 8

Show Influx Data in a Node-Red Dashboard

For a simple Dashboard I wanted to use a dropdown node (as a time selector), a chart and a table.

The drop down node has a selection of different times.

The payload from the dropdown node would be something like: 1m, 5m, 15m. A Javascript function node (“New Time Scale”) used this payload and created an InfluxDB query.

This syntax can be tested in the influx cli:

> select time,temperature from mytemps where location='Burlington' and time > now() - 5m
name: mytemps
time temperature
---- -----------
1580588829859372644 12
1580588889896729245 6
1580588949931621672 17
1580589009972333308 8
1580589069980649689 12

The InfluxDB input node only has the InfluxDB server information. The query is passed in from the Javascript function node (“New Time Scale”) .

A Javascript function node (“Javascript function node (“Format Influx Results”) is used to put the msg.payload into a format that the chart node can use.


//
// Format the InfluxDB results to match the charts JSON format
//

var series = ["temp DegC"];
var labels = ["Data Values"];
var data = "[[";
var thetime;

for (var i=0; i < msg.payload.length; i++) {
    thetime = Number(msg.payload[i].time); // Some manipulation of the time may be required
    data += '{ "x":' + thetime + ', "y":' + msg.payload[i].temperature + '}';
    if (i < (msg.payload.length - 1)) {
        data += ","
    } else {
        data += "]]"
    }
}
var jsondata = JSON.parse(data);
msg.payload = [{"series": series, "data": jsondata, "labels": labels}];
return msg;


Once all the logic has been updated, click on the Deploy button. The Node-Red dashboard can be accessed at: http://node-red_ip:1880/ui. Below is an example:

Final Comments
This project was not 100% there are still some cleanup items to do, such as:

use real I/O
make the times a little cleaner in the table
a better time selections for the chart.

Also to better explain things I only used 1 location but multiple data points could be inserted, queried and charted.

Sqlite and Node-Red

December 26, 2019May 27, 2021Posted in Node-RedTagged chart, Node-Red, NodeRed, Raspberry Pi, sqlite23 Comments

Sqlite is an extremely light weight database that does not run a server component.

In this blog I wanted to document how I used Node-Red to create, insert and view SQL data on a Raspberry Pi. I also wanted to show how to reformat the SQL output so that it could be viewed in a Node-Red Dashboard line chart.

Installation

Node-Red is pre-installed on the Pi Raspian image. I wasn’t able to install the Sqlite node using the Node-Red palette manager. Instead I did a manual install as per the directions at: https://flows.nodered.org/node/node-red-node-sqlite .

cd ~/.node-red npm i --unsafe-perm node-red-node-sqlite npm rebuild

Create a Database and Table

It is possible to create a database and table structures totally in Node-Red.

I connected a manual inject node to a sqlite node.

In the sqlite node an SQL create table command is used to make a new table. Note: the database file is automatically created.

For my example I used a 2 column table with a timestamp and a value

Insert Data into Sqlite

Data can be inserted into Sqlite a number of different ways. A good approach for a Rasp Pi is to pass some parameters into an SQL statement.

The sqlite node can use a “Prepared Statement” with a msg.params item to pass in data. For my example I created two variable $thetime and $thevalue.

A function node can be used to format a msg.params item.


// Create a Params variable
// with a time and value component
//
msg.params = { $thetime:Date.now(), $thevalue:msg.payload }
return msg;

Viewing Sqlite Data

A “select” statement is used in an sqlite node to view the data.

A simple SQL statement to get all the data for all the rows in this example would be:

select * from temps;

A debug node can used to view the output.

Custom Line Chart

Node-Red has a nice dashboard component that is well formatted for web pages on mobile devices.

To add the dashboard components use the Node-Red palette manager and search for: node-red-dashboard.

By default the chart node will create its own data vs. time storage. For many applications this is fine however if you want long term storage or customized historical plots then you will need to pass all the trend data to the chart node.

For some details on passing data into charts see: https://github.com/node-red/node-red-dashboard/blob/master/Charts.md#stored-data

Below is an example flow for creating a custom chart with 3 values with times.

The JavaScript code needs to create a structure with: series, data and labels definitions


msg.payload = [{
"series": ["A"],
"data": [
[{ "x": 1577229315152, "y": 5 },
{ "x": 1577229487133, "y": 4 },
{ "x": 1577232484872, "y": 6 }
]
],
"labels": ["Data Values"]
}];

return msg;

This will create a simple chart:

For reference, below is an example of the data structure for three I/O points with timestamps:


// Data Structure for: Three data points with timestamps

msg.payload = [{
"series": ["A", "B", "C"],
"data": [
[{ "x": 1577229315152, "y": 5 },
{ "x": 1577229487133, "y": 4 },
{ "x": 1577232484872, "y": 2 }
],
[{ "x": 1577229315152, "y": 8 },
{ "x": 1577229487133, "y": 2 },
{ "x": 1577232484872, "y": 11 }
],
[{ "x": 1577229315152, "y": 15 },
{ "x": 1577229487133, "y": 14 },
{ "x": 1577232484872, "y": 12 }
]
],
"labels": ["Data Values"]
}];

Sqlite Data in a Line Chart

To manually update a line chart with some Sqlite data I used the following nodes:

The SQL select statement will vary based on which time period or aggregate data is required. For the last 8 values I used:

select * from temps LIMIT 8 OFFSET (SELECT COUNT(*) FROM temps)-8;

The challenging part is to format the SQL output to match the required format for the Line Chart. You will need to iterate over each data row (payload object) and format a JSON string.

 //  
 // Create a data variable   
 //  
 var series = ["temp DegC"];  
 var labels = ["Data Values"];  
 var data = "[[";  
   
 for (var i=0; i < msg.payload.length; i++) {  
   data += '{ "x":' + msg.payload[i].thetime + ', "y":' + msg.payload[i].thetemp + '}';  
   if (i < (msg.payload.length - 1)) {  
     data += ","  
   } else {  
     data += "]]"  
   }  
 }  
 var jsondata = JSON.parse(data);  
 msg.payload = [{"series": series, "data": jsondata, "labels": labels}];  
   
   
 return msg;

To view the Node-Red Dashboard enter: http://pi_address:1880/ui

Final Comments

For a small standalone Raspberry Pi project using sqlite as a database is an excellent option. Because a Pi is limited in data storage I would need to include a function to limit the amount of data stored.

Apache Kafka with Node-Red

November 6, 2019November 9, 2019Posted in IoT, Messaging, Node-RedTagged dashboard, IoT, kafka, Node-Red, NodeRedLeave a comment

Apache Kafka is a distributed streaming and messaging system. There are a number of other excellent messaging systems such as RabbitMQ and MQTT. Where Kafka is being recognized is in the areas of high volume performance, clustering and reliability.

Like RabbitMQ and MQTT, Kafka messaging are defined as topics. Topics can be produced (published) and consumed (subscribed). Where Kafka differs is in the storage of messages. Kafka stores all produced topic messages up until a defined time out.

Node-Red is an open source visual programming tool that connects to Raspberry Pi hardware and it has web dashboards that can be used for Internet of Things presentations.

In this blog I would like to look at using Node-Red with Kafka for Internet of Things type of applications.

Getting Started

Kafka can be loaded on a variety of Unix platforms and Windows. A Java installation is required for Kafka to run, and it can be installed on an Ubuntu system by:

apt-get install default-jdk

For Kafka downloads and installation instructions see: https://kafka.apache.org/quickstart. Once the software is installed and running there a number of command line utilities in the Kafka bin directory that allow you to do some testing.

To test writing messages to a topic called iot_test1, use the kafka-console-producer.sh command and enter some data (use Control-C to exit):

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic iot_test1
11
22
33

To read back and listen for messages:

 bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic iot_test1 --from-beginning
11
22
33

The Kafka server is configured in the /config/server.properties file. A couple of the things that I tweeked in this file were:

# advertised the Kafka server node ip
advertised.listeners=PLAINTEXT://192.168.0.116:9092
# allow topics to be deleted
delete.topic.enable=true

Node-Red

Node-Red is a web browser based visual programming tool, that allows users to create logic by “wiring” node blocks together. Node-Red has a rich set of add-on components that includes things such as: Raspberry Pi hardware, Web Dash boards, email, Tweeter, SMS etc.

Node-Red has been pre-installed on Raspbian since 2015. For full installation instructions see: https://nodered.org/#get-started

To add a Node-Red component select the “Palette Manager”, and in the Install tab search for kafka. I found that the node-red-contrib-kafka-manager component to be reliable (but there are others to try).

For my test example I wanted to create a dashboard input that could be adjusted. Then read back the data from the Kafka server and show the result in a gauge.

This logic uses:

Kafka Consumer Group – to read a topic(s) from a Kafka server
Dashboard Gauge – to show the value
Dashboard Slider – allows a user to select a numeric number
Kafka Producer – sends a topic message to the Kafka server

Double-click on the Kafka nodes and in the ‘edit configuration’ dialog create and define a Kafka broker (or server). Also add the topic that you wish to read/write to.

Double-click on the gauge and slider nodes and define a Dashboard group. Also adjust the labels, range and sizing to meet your requirements.

After the logic is complete hit the Deploy button to run the logic. The web dashboard is available at: http://your_node_red_ip:1880/ui.

Final Comment

I found Node-Red and Kafka to be easy to use in a simple standalone environment. However when I tried to connect to a Cloud based Kafka service (https://www.cloudkarafka.com/) I quickly realized that there is a security component that needs to be defined in Node-Red. Depending on the cloud service that is used some serious testing will probably be required.

Raspberry Pi Internet Radio

November 4, 2019November 24, 2019Posted in Node-Red, Raspberry PiTagged internet radio, music, Node-Red, NodeRed, Pi, radio, Raspberry PiLeave a comment

Node-Red is graphical programming interface that allows logic to be created using nodes that are wired together. Node-Red has been pre-installed in the Raspian OS since 2015 and it has a very easy learning curve.

In this blog I wanted to show an example of Node-Red being used with a five button LCD faceplate to play Internet radio stations.

For this project I used a basic USB powered speaker, a Rasp Pi and a Pi 5-button LCD faceplate. The cost of the faceplates start at about $10.

Getting Started with Internet Radio

There are a good number of Internet radio resources, https://www.internet-radio.com has a good selection of stations to choose from.

To find a URL of a radio station, look through the stations until you find what you like and then right click on the .pls link, and “Save Link as…”. Save this link as a file and then open the file in a text editor to get the URL.

MPD – Music Player Daemon

MPD is a Linux based music service that supports the playing of both music files and internet based radio stations. For command line operations that is also a MPD client application called mpc. To install both the service and client:

sudo apt-get install mpd mpc

Before I started building the node-red application I played with the mpc commands to ensure that I understood the basics.

Internet radio stations are added like a song list:

mpc add 'http://uk2.internet-radio.com:8062'
mpc add 'http://live.leanstream.co/CKNXFM'
mpc add 'http://66.85.88.2:7136'

Some key mpc control commands are:

mpc play  # play the current station
mpc play 3 # play radio station 3
mpc pause  # pause the music
mpc next  # play the next radio station
mpc prev  # play the previous radio station 

mpc volume 90 # set the volume to 90%
mpc volume +5 # increase the volume 5%
mpc volume -5 # decrease the volume 5%

The mpc status command will show the volume, what is playing along with the current station number and total number of stations:

$ mpc status
Comedy104 - A Star104.net Station: Doug Stanhope - To Tell You the Truth
[playing] #2/4 1:45/0:00 (0%)
volume: 75% repeat: off random: off single: off consume: off

Node-Red Logic

Node-Red can be started from the Raspberry Pi menus, or from the command line:

node-red-start &

To access the Node-Red web page, enter the Raspberry Pi ip address with port 1880, for example : http://192.168.0.121:1880

For this project two extra Node-Red components are needed, and they are for the LCD faceplate and the MPD music player. To add components use the “Palette Manager” option.

For the LCD faceplate, search for Adafruit, and select the i2c-lcd-adafruit-sainsmart component.

Similarly search for mpd and add the node-red-contrib-mpd component.

To create logic select a node from the left node panel and drag it onto the center flow palette, and then “wire” the nodes together.

For the Internet music example I used four function nodes, and the two i2cLED and the two MPD nodes. (Comment nodes are only used to explain the logic).

The first step is to double click on the MPD nodes and add an MPD server.

Select Button Logic

I used the select button to turn on and off the music player.

A context variable is created to save the current state of the player. Note: a context variable is only accessible for the node where it is defined..

The ic2_LCD_Input node message has a msg.button_name and msg.button_state item that is used to determine which button is pushed.

For the select button logic a group of messages was used to add the different radio stations.


// create an "is setup" variable
var issetup = context.get('issetup')||0;

if ((msg.button_name == "SELECT") && (msg.button_state == "pressed" )){
// if the setup hasn't been run, add a radio station playlist
if (issetup === 0) {
context.set('issetup',1);
var msg0 = { payload:"clear"};
var msg1 = { payload:"add http://185.33.21.112:11029" }; // 1.FM Trance
var msg2 = { payload:"add http://66.85.88.2:7136" }; // Comedy 104
var msg3 = { payload:"add http://live.leanstream.co/CKNXFM"}; // The One - Wingham
var msg4 = { payload:"add http://185.33.21.112:11269" }; // Baroque
var msg5 = { payload:"play" };
return [ [ msg0, msg1, msg2, msg3, msg4, msg5] ];
} else {
context.set('issetup',0);
msg0 = { payload:"pause"};
return msg0;
}
}

Up/Down Button Logic

The UP button will issue an MPD command equivalent to :

mpc volume +5

This will up the volume by 5%. The total volume will max at 100%.

The DOWN button will issue an MPD command equivalent to :

mpc volume -5


// Raise/Lower the volume
var msg1;
var thevolume = 5; //volume % increment to change

if ((msg.button_name == "UP") && (msg.button_state == "pressed" )){
// if the setup hasn't been run, add a radio station playlist
msg1 = { payload:"volume +" + thevolume };
return msg1;
}
if ((msg.button_name == "DOWN") && (msg.button_state == "pressed" )){
// if the setup hasn't been run, add a radio station playlist
msg1 = { payload:"volume -" + thevolume};
return msg1;
}

Current and Max Radio Station Logic

The ‘Current and Max Radio Stations’ node is updated from the MPD in node when there are any changes to the volume or when a new song or station is played.

This logic creates two flow variables (stnmax, stncnt) that are available in any node in this flow. The station max (stnmax) and current radio station (stncnt) variables are used in the LEFT/RIGHT button logic to determine which station to change to.


// Get the max number of radio stations and the current radio statio
// Make context variables that can be used in other node, like the LEFT/RIGHT button

var msg1 = msg.payload.status ; //create a simplier message
var stnmax = msg1.playlistlength;
flow.set('stnmax',stnmax);
var stncur = msg1.nextsong;
if (isNaN(stncur)) {stncur = stnmax;} // ensure a valid station

flow.set('stncur',stncur);

return msg1; // only needed for possible debugging

While the code is running it is possible to view the context date.

UP/DOWN Button Logic

The UP / DOWN logic changes between the radio stations using the mpc commands:

mpc next
mpc prev

It is important to not move past the range of the radio stations or MPD will hang. The stnmax and stncur variables are used to determine if the next or previous commands are to be allowed.


// Move left and right in radion stations
var stnmax = flow.get('stnmax');
var stncur = flow.get('stncur');
if ((msg.button_name == "LEFT") && (msg.button_state == "pressed" )){
// if the setup hasn't been run, add a radio station playlist
if (stncur > 1) {
var msg0 = {payload:"previous"};
return msg0;
}
}
if ((msg.button_name == "RIGHT") && (msg.button_state == "pressed" )){
// if the setup hasn't been run, add a radio station playlist
if (stncur < stnmax)
var msg1 = {payload:"next"};
return msg1;

}

Final Comments

The Pi LCD faceplate is an excellent hardware add-on for any Raspberry Pi project. However it important to know that clone hardware may work as expected. For my hardware I was not able to easily turn off the extra LED.

A future enhancement would be to add a Web interface so that you could change the volume or stations without using the 5 button Pi faceplate.

ODROID – A Raspberry Pi Alternative

March 30, 2019March 30, 2019Posted in UncategorizedTagged C, Node-Red, NodeRed, Odroid, Python, Raspberry PiLeave a comment

The ODROID is a series of single-board computers manufactured by Hardkernel in South Korea. The ODROID-C1+ ($35) and the ODROID-C2 ($46) have a form factor similar to the Raspberry Pi 3. The higher end ODROID-XU($59) which is around 5 times faster than the Pi 3 has a signifigently different board layout.

For my testing I looked at the ODROID-C2 ($46), it is a little more expensive than a Pi 3 but the literature states that it’s 2-3 times faster.

My goal was to see if I could use the ODROID-C2 for some typical Raspberry Pi applications. In this blog I will be looking at doing C, Python and NodeRed programming from a Pi user perspective.

I’ve been happy with the functionality and openness of the Raspberry Pi platform, however I find its desktop performance to be sluggish. For only a few dollars more than a Pi 3 the ODROID-C2 CPU, RAM and GPU specs are impressive.

	ODROID-C2 / Raspberry Pi 3 Hardware Comparison

	Odroid C2	Raspberry Pi 3
CPU	Amlogic S905 SoC 4 x ARM Cortex-A53 1.5GHz 64bit ARMv8 Architecture @28nm	Broadcom BCM2837 4 x ARM Cortex-A53 1.2Ghz 64bit ARMv7 Architecture @40nm
RAM	2GB 32bit DDR3 912MHz	1GB 32bit LPDDR2 450MHz
GPU	3 x ARM Mali-450 MP 700MHz	1 x VideoCore IV 250MHz
USB	4 Ports	4 Ports
Ethernet / LAN	10 / 100 / 1000 Mbit/s	10 / 100 Mbit/s
Built in WiFi	No	Yes
Built in Bluetooth	No	Yes
IR Receiver	Built in	Needs add-on
I/O Expansion	40 + 7 pin port GPIO / UART / I2C / I2S / ADC	40 pin port GPIO / UART / SPI / I2S
Camera Input	USB 720p	MIPI CSI 1080p
List Price (US)	$46	$35

First Impressions

The ODROID-C2 is almost the same footprint as the Raspberry Pi 3 but not exactly. I found that because the microSD mounting is different some, but not all, of my Pi cases could be used .

When you are designing your projects it is important to note that the ODROID-C2 does not have a built in Wifi or Bluetooth adapters, so you’ll need wired connections or USB adapters. Like some of the Orange Pi modules the ODROID-C2 has a built-in IR connection.

ODROID-C2 can be loaded with Ubuntu, Arch Linux and Android images. For my testing I used the Armbian 5.40 Ubuntu desktop and the performance was signifigently faster than my Raspberry Pi 3 Raspian desktop. I could definitely see ODROID-C2 being used as a low cost Web browser station.

The ODROID-C2 images are quite lean, so you will need to go the ODROID Wiki, https://wiki.odroid.com, for instructions on loading additional software components.

The ODROID-C2 has a 40 pin General Purpose Input/Output (GPIO) arrangement like the Pi 3, so it is possible to use Pi prototyping hats on the ODROID-C2 .

There are some noticeable differences in the pin definitions between the two modules, so for this reason I didn’t risk using any of my intelligent Pi hats on the ODROID-C2. The gpio command line tool can be used to view the pin definitions:

The Raspberry Pi GPIO names are in the range of 2 to 27, whereas the ODROID-C2 GPIO ranges are in the 200’s, because of this don’t expect to be able to run all your Raspberry Pi code “as is” on the ODROID-C2.

Unlike the Arduino the Raspberry Pi platform has no built in support for analog inputs. I got pretty excited when I noticed that the ODROID-C2 had two built in Analog-to-Digital Converter (ADC) pins (AIN.1 on pin 37 and AIN.0 on pin 40). However after some investigation I found that these pins had virtually no example code and they only support 1.8 volts. Most of my analog input sensors require 3.3V or 5V so I’m not sure how often these ADC pins will be used.

Python Applications

The ODROID-C2 Wiki references the RPi.GPIO and wiringpi Python libraries. I tested both of these libraries and I found that standard reads and writes worked, but neither of these libraries supported the callback functions like the Raspberry Pi versions.

For existing Pi projects where you are using callback functions for rising and/or falling digital signals (like intrusion alarms) you will need to do some re-coding with a polling method. It’s also important to note that the ODROID RPi.GPIO library is a little confusing because it uses the Pi pin names and not the ODROID pin names, so for example ODROID-C2 physical pin 7 is referenced as GPIO.04 (as on a PI) and not GPIO.249 (the ODROID-C2 name). Below is simple Python example that polls for a button press and then toggles an LED output when a button press is caught.

import RPi.GPIO as GPIO

GPIO.setmode(GPIO.BCM)

button = 4 # physical pin 7, PI GPIO.04, ODROID-C2 GPIO.249
led = 17 # physical pin 11, PI GPIO.17, ODROID-C2 GPIO.247
GPIO.setup(led, GPIO.OUT, initial=GPIO.LOW)
GPIO.setup(button, GPIO.IN, pull_up_down = GPIO.PUD_UP)

print ('Wait for button...')
while True:
    if GPIO.input(button) == 1:
        GPIO.output(led,0)
    else:
    GPIO.output(led,1)
    print "button pressed"

There are some excellent Python libraries that are designed to work with the Raspberry Pi. However it will require some trial and error to determine which libraries will and won’t work with the ODROID-C2.

I tried testing the DHT11 temperature and humidity sensor with the ODROID-C2, and unfortunately the library had a “Segmentation Error” when I tried running an example.

Node-Red

NodeRed can be installed on ODROID-C2 by using the manual install instructions for Raspbian at nodered.org. This install procedure will add a start item to the desktop Application menu, but due to hardware differences the Raspberry Pi GPIO input/output components will not load.

To read and write to Raspberry Pi GPIO a simple workaround is to use exec nodes to call the gpio utility.

The command line syntax for writing a gpio output is: gpio write pin state, and for reading it is: gpio read pin. One of the limitations of this workaround is that you will need to add a polling mechanism, luckily there are some good scheduling nodes such as Big Timer that can be used.

C Applications

Programming in C is fairly well documented and an ODROID “C Tinkering Kit” is sold separately. The wiringPi library is used in C applications.

Below is a C version of the Python example above. It is important to note that these 2 examples talk to the same physical pins but the C wiringPi library uses the ODROID-C2 wPi numbers and the Python RPi.GPIO library uses the Pi BCM numbers.

// led.c - Read/Write "C" example for an Odroid-C2
//
#include  <wiringPi.h>

int main(void)
{
    wiringPiSetup();
    int led = 0;
    int button = 7;
    pinMode(led, OUTPUT);
    pinMode(button, INPUT);
 
    for (;;)
    {
        if (digitalRead(button) == 1) {
           digitalWrite(led, LOW); 
        }
	else
	{
           digitalWrite(led, HIGH); 
        }
    }
    return 0;
}

To compile and run this program:

$ gcc -o led led.c -lwiringPi -lpthread
$ sudo ./led

Summary

I liked that I could reuse some of my Pi cases and prototyping hats with the ODROID-C2.

As a PI user I found that coding in C, Python and NodeRed on the ODROID-C2 was fairly easy, but there were many limitation compared to the Pi platform. The ODROID Wiki had the key product documentation, but it was no where near the incredibly rich documentation that exists with Raspberry Pi modules.

There are a some excellent Python libraries and PI hardware add-ons that support a variety of sensors and I/O applications, these may or not work with the ODROID hardware.

During the development cycle of a project it is nice to have a faster interface, but typically my final projects do not need any high end processing and they can often run on low end Raspberry Pi 1 modules. So for now I would stick to a Raspberry Pi for GPIO/hardware type projects.

I enjoying playing with the ODROID-C2 and for projects requiring higher performance such as video servers, graphic or Web applications then the ODROID-C2 module is definitely worth considering.

IoT with Google Firebase

October 9, 2018November 9, 2019Posted in IoT, MessagingTagged Firebase, Google, Node-Red, NodeRed, Python, Raspberry Pi7 Comments

For many Internet of Things (IoT) projects a message queuing system like MQTT (Message Queue Telemetry Transport) is all that is needed to connect sensors, devices and graphic interfaces together. If however you require a database, with sorting, queuing and multi-media support then there are some great cloud storage platforms that are available. One that is definitely worth taking a look at is Google Firebase.

Like any IoT solution, Google Firebase can have inputs and sensors sending data directly into it, and a variety of client applications to view the data (Figure 1), but Google Firebase also offers other features such as: file storage, machine learning, messaging, and server side functions. In this article I will look at:

In this article I will look at:

Setting up a sample Firebase IoT project
Use Python to simulate I/O data
Create a Web dashboard with Node-Red to view and write data
Create an Android app using AppInventor to view and write data
Look at a more complex data monitoring example in Python

Getting Started

Google Firebase [1] may not have the huge variety of options that Amazon Web Services (AWS) has, but I found as an IoT engineer Google Firebase had all the features that I needed to get up and running quickly, and I didn’t need a credit card for my free activation.

To begin log in with your Google account at https://firebase.google.com/, and select “Get Started”. I created a sample project called My-IoT. After the project is created Firebase will give it a unique identifier name, such as: my-iot-4ded7.

Firebase data can be stored in either a Realtime Database, or a Cloud Firestore. The Realtime Database is an unstructured NoSQL format that has been around for about 4 years, so there are lots of 3rd party components. The Cloud Firestore stores the data in structure collections and it’s fairly new so the 3rd party options are a little more limited.

For our test IoT project we will use the real-time database, and it can be created from the Database -> Data menu option.

The database structure can be built directly from the Firebase web console or imported from a JSON file. For this database 3 grouping are defined, one for Android, Node-Red and Raspberry Pi I/O values. The Firebase console allows for setting, changing and viewing the database values.

The security is configured in the Database -> Rules menu option. To keep things simple for testing read and write security is set to true for public access, (remember to change this for a production system).

The project’s remote access settings are shown in the Authentication section with the “Web setup” button.

Python and Firebase

There are a number of Python libraries to access Firebase. The pyrebase library has some added functionality such as: queries, sorting, file downloads and streaming support. The pyrebase library only support Python 3, and it is installed by:

sudo pip3 install pyrebase
sudo pip3 install --upgrade google-auth-oauthlib

A Python 3 test program to set two values (pi/pi_value1 and pi/pi_value2) and read a single point and a group of point is shown below. Remember to change the configuration settings to match your project settings.

import pyrebase, random

# define the Firebase as per your settings
config = {
  "apiKey": "AIzaSyCq_WytLdmOy1AIzaSyCq_WytLdmOy1",
  "authDomain": "my-iot-4ded7.firebaseapp.com",
  "databaseURL": "https://my-iot-4ded7.firebaseio.com",
  "storageBucket": "my-iot-4ded7.appspot.com"
}

firebase = pyrebase.initialize_app(config)
db = firebase.database()

# set 2 values with random numbers
db.child("pi").child("pi_value1").set(random.randint(0,100))
db.child("pi").child("pi_value2").set(random.randint(0,100))

# readback a single value
thevalue = db.child("pi").child("pi_value1").get().val()
print ("Pi Value 1: ", thevalue)

# get all android values
all_users = db.child("android").get()
for user in all_users.each():
    print(user.key(), user.val())

The Firebase web console can be used to check the results from the test program.

Node-Red and Firebase

Node-Red is a visual programming environment that allows users to create applications by dragging and dropping nodes on the screen. Logic flows are then created by connecting the different nodes together.

Node-Red has been preinstalled on Raspbian Jesse since the November 2015. Node-Red can also be installed on Windows, Linux and OSX. To install and run Node-Red on your specific system see https://nodered.org/docs/getting-started/installation.

To install the Firebase components, select the Manage palette option from the right side of the menu bar. Then search for “firebase” and install node-red-contrib-firebase.

For our Node-Red example we will use web gauges to show the values of our two Python simulated test points. We will also send a text comment back to our Firebase database. The complete Node-Red logic for this is done in only 6 nodes!

To read the Pi values two firebase.on() nodes are used. The output from these nodes is connected to two dashboard gauge nodes. Double-clicking on firebase_on() node to configure Firebase database and the item to read from. Double-clicking on the gauge node allows you to edit gauge properties.

To send a text string to Firebase a text input node is wired to a firebase modify node. Edit the firebase modify node with the correct database address and value to set.

After the logic is complete, hit the Deploy button on the right side of the menu bar to run the logic. The Node-Red dashboard user interface is accessed by: http://ipaddress:1880/ui, so for example 192.168.1.108:1880/ui.

AppInventor

AppInventor is a Web based Android app creation tool (http://appinventor.mit.edu), that uses a graphical programming environment.

AppInventor has two main screens. The Designer screen is used for the layout or presentation of the Android app and the Blocks screen is used to build the logic. On the right side of the top menu bar, the Designer and Blocks buttons allow you to toggle between these two screens.

On the Designer screen, an app layout is created by dragging a component from the Palette window onto the Viewer window.

We will use AppInventor to create an Android app that will read Pi values from our Firebase IoT database, and write a value back.

For the visuals on this application a Button, Label (2), ListView, Textbox and FirebaseDB component will be used. After a component is added to the application, the Components window is used to rename or delete that component. The Properties window is used to edit the features on a component. For the FirebaseDB component configure the Token and URL to match with your project settings.

Once the layout design is complete, logic can be added by clicking on the Blocks button on the top menu bar.

Logic is built by selecting an object in the Blocks window, and then click on the specific block that you would like to use.

AppInventor is pretty amazing when it comes to doing some very quick prototyping. Our entire app only uses two main blocks.

The when FirebaseDB1.DataChanged block is executed whenever new data arrives into the Firebase database. The DataChanged block returns a tag and a value variable. The tag variable is the top level item name, (“pi” or “nodered” for our example). The value will be a string of the items and their values, for example: “{pi_value2 = 34, pi_value1=77}”. We use a replace all text block to remove the “{” and “}” characters, then we pass the string to the ListView component. Note this same code will pass 2 or 200 tags in the pi tag section.

The when BT_PUT.click block will pass the text that we enter on the screen into the android/and_value1 item in our database.

After the screen layout and logic is complete, the menu item Build will compile the app. The app can be made available as an APK downloadable file or as a QR code link.

Below is picture of the final Android app synced with the Firebase data.

Python Data Monitoring Example

Our starting IoT Firebase realtime database was quite simple. Data monitoring or SCADA (Supervisory Control and Data Aquisition) projects usually require more information than a just a value. A more realistic sensor database would includes fields such as tag name, description, status, time and units.

By adding some indexing in the Firebase Security Rules it is possible to create some queries and sorts of the data. Some typical queries would be: “Points in alarm” or “Point values between 2:00 and 2:15”.

In the code listing below the syntax statement of .order_by_child(“status”).equal_to(“ALARM”).get() is used to show only records that have a status = ALARM. Some other filter options include: .start_at(time1).end_at(time2).get, .limit_to_first(n), and .order_by_value().

A good next step would be to pass important filtered information to Google Firebase’s messaging feature.

import pyrebase, random

# define the Firebase as per your settings
config = {
  "apiKey": "AIzaSyCq_AIzaSyCq_Wyt",
  "authDomain": "my-iot-7ded7.firebaseapp.com",
  "databaseURL": "https://my-iot-7ded7.firebaseio.com",
  "storageBucket": "my-iot-7ded7.appspot.com"
}

firebase = pyrebase.initialize_app(config)
db = firebase.database()

tag_sum = db.child("pi").order_by_child("status").equal_to("ALARM").get()

# print alarm points
for tag in tag_sum.each():
    info = tag.val()
    print (tag.key()," - ", info["description"],":", info["value"],info["units"], info["status"])

For more Python examples see the Pyrebase documentation.

Summary

Without a lot of configuration it is possible to configure a Google Firebase project and have Python, Node-Red and Android app read and write values. The level of programming complexity is on par with an MQTT implementation, however Google Firebase can offer a lot more future functionality that you wouldn’t have with MQTT, such as file storage, machine learning, messaging, and server side functions.

At the time of writing this article I found the Arduino Firebase library to be unreliable but Google offers some options like an MQTT bridge.