Apache Kafka with Node-Red

Apache Kafka is a distributed streaming and messaging system. There are a number of other excellent messaging systems such as RabbitMQ and MQTT. Where Kafka is being recognized is in the areas of high volume performance, clustering and reliability.

Like RabbitMQ and MQTT, Kafka messaging are defined as topics. Topics can be produced (published) and consumed (subscribed). Where Kafka differs is in the storage of messages. Kafka stores all produced topic messages up until a defined time out.

Node-Red is an open source visual programming tool that connects to Raspberry Pi hardware and it has web dashboards that can be used for Internet of Things presentations.

In this blog I would like to look at using Node-Red with Kafka for Internet of Things type of applications.

Getting Started

Kafka can be loaded on a variety of Unix platforms and Windows.  A Java installation is required for Kafka to run, and it can be installed on an Ubuntu system by:

apt-get install default-jdk

For Kafka downloads and installation instructions see: https://kafka.apache.org/quickstart. Once the software is installed and running there a number of command line utilities in the Kafka bin directory that allow you to do some testing.

To test writing messages to a topic called iot_test1, use the kafka-console-producer.sh  command and enter some data (use Control-C to exit):

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic iot_test1
11
22
33

To read back and listen for messages:

 bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic iot_test1 --from-beginning
11
22
33

The Kafka server is configured in the /config/server.properties  file. A couple of the things that I tweeked in this file were:

# advertised the Kafka server node ip
advertised.listeners=PLAINTEXT://192.168.0.116:9092
# allow topics to be deleted
delete.topic.enable=true

Node-Red

Node-Red is a web browser based visual programming tool, that allows users to create logic by “wiring” node blocks together.  Node-Red has a rich set of add-on components that includes things such as: Raspberry Pi hardware, Web Dash boards, email, Tweeter, SMS etc.

Node-Red has been pre-installed on Raspbian since 2015. For full installation instructions see:  https://nodered.org/#get-started

To add a Node-Red component select the “Palette Manager”, and in the Install tab search for kafka. I found that the node-red-contrib-kafka-manager component to be reliable (but there are others to try).

For my test example I wanted to create a dashboard input that could be adjusted. Then read back the data from the Kafka server and show the result in a gauge.

This logic uses:

  • Kafka Consumer Group – to read a topic(s) from a Kafka server
  • Dashboard Gauge – to show the value
  • Dashboard Slider – allows a user to select a numeric number
  • Kafka Producer – sends a topic message to the Kafka server

nodered_kafka Double-click on the Kafka nodes and in the ‘edit configuration’ dialog create and define a Kafka broker (or server). Also add the topic that you wish to read/write to.

kafka_consume

Double-click on the gauge and slider nodes and define a Dashboard group. Also adjust the labels, range and sizing to meet your requirements.

kafka_gauge

After the logic is complete hit the Deploy button to run the logic. The web dashboard is available at: http://your_node_red_ip:1880/ui.

kafka_phone

Final Comment

I found Node-Red and Kafka to be easy to use in a simple standalone environment. However when I tried to connect to a Cloud based Kafka service (https://www.cloudkarafka.com/) I quickly realized that there is a security component that needs to be defined in Node-Red. Depending on the cloud service that is used some serious testing will probably be required.

 

Pi Node-Red Internet Radio

Node-Red is graphical programming interface that allows logic to be created using nodes that are wired together. Node-Red has been pre-installed in the Raspian OS since 2015 and it has a very easy learning curve.

In this blog I wanted to show an example of Node-Red being used with a five button LCD faceplate to play Internet radio stations.

For this project I used a basic USB powered speaker and a Raspberry Pi 5-button LCD faceplate. The cost of the faceplates start at about $10.

Getting Started with Internet Radio

There are a good number of Internet radio resources, https://www.internet-radio.com has a good selection of stations to choose from.

To find a URL of a radio station, look through the stations until you find what you like and then right click on the .pls link, and “Save Link as…”. Save this link as a file and then open the file in a text editor to get the URL.

radio_stations

 

MPD – Music Player Daemon

MPD is a Linux based music service that supports the playing of both music files and internet based radio stations. For command line operations that is also a MPD client application called mpc. To install both the service and client:

sudo apt-get install mpd mpc

Before I started building the node-red application I played with the mpc commands to ensure that I understood the basics.

Internet radio stations are added like a song list:

mpc add 'http://uk2.internet-radio.com:8062'
mpc add 'http://live.leanstream.co/CKNXFM'
mpc add 'http://66.85.88.2:7136'

Some key mpc control commands are:

mpc play  # play the current station
mpc play 3 # play radio station 3
mpc pause  # pause the music
mpc next  # play the next radio station
mpc prev  # play the previous radio station 

mpc volume 90 # set the volume to 90%
mpc volume +5 # increase the volume 5%
mpc volume -5 # decrease the volume 5%

The mpc status command will show the volume, what is playing along with the current station number and total number of stations:

$ mpc status
Comedy104 - A Star104.net Station: Doug Stanhope - To Tell You the Truth
[playing] #2/4 1:45/0:00 (0%)
volume: 75% repeat: off random: off single: off consume: off

Node-Red Logic

Node-Red can be started from the Raspberry Pi menus, or from the command line:

node-red-start &

To access the Node-Red web page, enter the Raspberry Pi ip address with port 1880, for example : http://192.168.0.121:1880

For this project two extra Node-Red components are needed, and they are for the LCD faceplate and the MPD music player. To add components use the “Palette Manager” option.

palette

For the LCD faceplate, search for Adafruit, and select the i2c-lcd-adafruit-sainsmart component.

adafruit_palette

Similarly search for mpd and add the node-red-contrib-mpd component.

mpd_palette To create logic select a node from the left node panel and drag it onto the center flow palette, and then “wire” the nodes together.

For the Internet music example I used four function nodes, and the two i2cLED and the two MPD nodes. (Comment nodes are only used to explain the logic).

node_red_radio

The first step is to double click on the MPD nodes and add an MPD server.

Select Button Logic

I used the select button to turn on and off the music player.

A context variable is created to save the current state of the player. Note: a context variable is only accessible for the node where it is defined..

The ic2_LCD_Input node message has a msg.button_name and msg.button_state item that is used to determine which button is pushed.

For the select button logic a group of messages was used to add the different radio stations.


// create an "is setup" variable
var issetup = context.get('issetup')||0;

if ((msg.button_name == "SELECT") && (msg.button_state == "pressed" )){
// if the setup hasn't been run, add a radio station playlist
if (issetup === 0) {
context.set('issetup',1);
var msg0 = { payload:"clear"};
var msg1 = { payload:"add http://185.33.21.112:11029" }; // 1.FM Trance
var msg2 = { payload:"add http://66.85.88.2:7136" }; // Comedy 104
var msg3 = { payload:"add http://live.leanstream.co/CKNXFM"}; // The One - Wingham
var msg4 = { payload:"add http://185.33.21.112:11269" }; // Baroque
var msg5 = { payload:"play" };
return [ [ msg0, msg1, msg2, msg3, msg4, msg5] ];
} else {
context.set('issetup',0);
msg0 = { payload:"pause"};
return msg0;
}
}

Up/Down Button Logic

The UP button will issue an MPD command equivalent to :

mpc volume +5

This will up the volume by 5%. The total volume will max at 100%.

The DOWN button will issue an MPD command equivalent to :

mpc volume -5

// Raise/Lower the volume
var msg1;
var thevolume = 5; //volume % increment to change

if ((msg.button_name == "UP") && (msg.button_state == "pressed" )){
// if the setup hasn't been run, add a radio station playlist
msg1 = { payload:"volume +" + thevolume };
return msg1;
}
if ((msg.button_name == "DOWN") && (msg.button_state == "pressed" )){
// if the setup hasn't been run, add a radio station playlist
msg1 = { payload:"volume -" + thevolume};
return msg1;
}

Current and Max Radio Station Logic

The ‘Current and Max Radio Stations’ node is updated from the MPD in node when there are any changes to the volume or when a new song or station is played.

This logic creates two flow variables (stnmax, stncnt) that are available in any node in this flow.  The station max (stnmax) and current radio station (stncnt) variables are used in the LEFT/RIGHT button logic to determine which station to change to.


// Get the max number of radio stations and the current radio statio
// Make context variables that can be used in other node, like the LEFT/RIGHT button

var msg1 = msg.payload.status ; //create a simplier message
var stnmax = msg1.playlistlength;
flow.set('stnmax',stnmax);
var stncur = msg1.nextsong;
if (isNaN(stncur)) {stncur = stnmax;} // ensure a valid station

flow.set('stncur',stncur);

return msg1; // only needed for possible debugging

While the code is running it is possible to view the context date.

context_flow

UP/DOWN Button Logic

The UP / DOWN logic changes between the radio stations using the mpc commands:

mpc next
mpc prev

It is important to not move past the range of the radio stations or MPD will hang. The stnmax and stncur variables are used to determine if the next or previous commands are to be allowed.


// Move left and right in radion stations
var stnmax = flow.get('stnmax');
var stncur = flow.get('stncur');
if ((msg.button_name == "LEFT") && (msg.button_state == "pressed" )){
// if the setup hasn't been run, add a radio station playlist
if (stncur > 1) {
var msg0 = {payload:"previous"};
return msg0;
}
}
if ((msg.button_name == "RIGHT") && (msg.button_state == "pressed" )){
// if the setup hasn't been run, add a radio station playlist
if (stncur < stnmax)
var msg1 = {payload:"next"};
return msg1;

}

Final Comments

The Pi LCD faceplate is an excellent hardware add-on for any Raspberry Pi project. However it important to know that clone hardware may work as expected. For my hardware I was not able to easily turn off the extra LED.

A future enhancement would be to add a Web interface so that you could change the volume or stations without using the 5 button Pi faceplate.

 

ODROID – A Raspberry Pi Alternative

The ODROID is a series of single-board computers manufactured by Hardkernel in South Korea. The ODROID-C1+ ($35) and the ODROID-C2 ($46) have a form factor similar to the Raspberry Pi 3. The higher end ODROID-XU($59) which is around 5 times faster than the Pi 3 has a signifigently different board layout.

For my testing I looked at the ODROID-C2 ($46), it is a little more expensive than a Pi 3 but the literature states that it’s 2-3 times faster.

My goal was to see if I could use the ODROID-C2 for some typical Raspberry Pi applications. In this blog I will be looking at doing C, Python and NodeRed programming from a Pi user perspective.

OD_PI

I’ve been happy with the functionality and openness of the Raspberry Pi platform, however I find its desktop performance to be sluggish. For only a few dollars more than a Pi 3 the ODROID-C2 CPU, RAM and GPU specs are impressive.

ODROID-C2 / Raspberry Pi 3 Hardware Comparison
Odroid C2 Raspberry Pi 3
CPU Amlogic S905 SoC
4 x ARM Cortex-A53 1.5GHz
64bit ARMv8 Architecture @28nm
Broadcom BCM2837
4 x ARM Cortex-A53 1.2Ghz
64bit ARMv7 Architecture @40nm
RAM 2GB 32bit DDR3 912MHz 1GB 32bit LPDDR2 450MHz
GPU 3 x ARM Mali-450 MP 700MHz 1 x VideoCore IV 250MHz
USB 4 Ports 4 Ports
Ethernet / LAN 10 / 100 / 1000 Mbit/s 10 / 100 Mbit/s
Built in WiFi No Yes
Built in Bluetooth No Yes
IR Receiver Built in Needs add-on
I/O Expansion 40 + 7 pin port
GPIO / UART / I2C / I2S / ADC
40 pin port
GPIO / UART / SPI / I2S
Camera Input USB 720p MIPI CSI 1080p
List Price (US) $46 $35

First Impressions

The ODROID-C2 is almost the same footprint as the Raspberry Pi 3 but not exactly. I found that because the microSD mounting is different some, but not all, of my Pi cases could be used .

lego_case

When you are designing your projects it is important to note that the ODROID-C2 does not have a built in Wifi or Bluetooth adapters, so you’ll need wired connections or USB adapters. Like some of the Orange Pi modules the ODROID-C2 has a built-in IR connection.

ODROID-C2 can be loaded with Ubuntu, Arch Linux and Android images. For my testing I used the Armbian 5.40 Ubuntu desktop and the performance was signifigently faster than my Raspberry Pi 3 Raspian desktop. I could definitely see ODROID-C2 being used as a low cost Web browser station.

The ODROID-C2 images are quite lean, so you will need to go the ODROID Wiki, https://wiki.odroid.com, for instructions on loading additional software components.

The ODROID-C2 has a 40 pin General Purpose Input/Output (GPIO) arrangement like the Pi 3, so it is possible to use Pi prototyping hats on the ODROID-C2 .

pi_hats

There are some noticeable differences in the pin definitions between the two modules, so for this reason I didn’t risk using any of my intelligent Pi hats on the ODROID-C2. The gpio command line tool can be used to view the pin definitions:

Od_readall

The Raspberry Pi GPIO names are in the range of 2 to 27, whereas the ODROID-C2 GPIO ranges are in the 200’s, because of this don’t expect to be able to run all your Raspberry Pi code “as is” on the ODROID-C2.

Unlike the Arduino the Raspberry Pi platform has no built in support for analog inputs. I got pretty excited when I noticed that the ODROID-C2 had two built in Analog-to-Digital Converter (ADC) pins (AIN.1 on pin 37 and AIN.0 on pin 40). However after some investigation I found that these pins had virtually no example code and they only support 1.8 volts. Most of my analog input sensors require 3.3V or 5V so I’m not sure how often these ADC pins will be used.

Python Applications

The ODROID-C2 Wiki references the RPi.GPIO and wiringpi Python libraries. I tested both of these libraries and I found that standard reads and writes worked, but neither of these libraries supported the callback functions like the Raspberry Pi versions.

For existing Pi projects where you are using callback functions for rising and/or falling digital signals (like intrusion alarms) you will need to do some re-coding with a polling method. It’s also important to note that the ODROID RPi.GPIO library is a little confusing because it uses the Pi pin names and not the ODROID pin names, so for example ODROID-C2 physical pin 7 is referenced as GPIO.04 (as on a PI) and not GPIO.249 (the ODROID-C2 name). Below is simple Python example that polls for a button press and then toggles an LED output when a button press is caught.

import RPi.GPIO as GPIO

GPIO.setmode(GPIO.BCM)

button = 4 # physical pin 7, PI GPIO.04, ODROID-C2 GPIO.249
led = 17 # physical pin 11, PI GPIO.17, ODROID-C2 GPIO.247
GPIO.setup(led, GPIO.OUT, initial=GPIO.LOW)
GPIO.setup(button, GPIO.IN, pull_up_down = GPIO.PUD_UP)

print ('Wait for button...')
while True:
    if GPIO.input(button) == 1:
        GPIO.output(led,0)
    else:
    GPIO.output(led,1)
    print "button pressed"

There are some excellent Python libraries that are designed to work with the Raspberry Pi. However it will require some trial and error to determine which libraries will and won’t work with the ODROID-C2.

I tried testing the DHT11 temperature and humidity sensor with the ODROID-C2, and unfortunately the library had a “Segmentation Error” when I tried running an example.

Node-Red

NodeRed can be installed on ODROID-C2 by using the manual install instructions for Raspbian at nodered.org. This install procedure will add a start item to the desktop Application menu, but due to hardware differences the Raspberry Pi GPIO input/output components will not load.

To read and write to Raspberry Pi GPIO a simple workaround is to use exec nodes to call the gpio utility.

Odroid_NodeRed

The command line syntax for writing a gpio output is: gpio write pin state, and for reading it is: gpio read pin. One of the limitations of this workaround is that you will need to add a polling mechanism, luckily there are some good scheduling nodes such as Big Timer that can be used.

C Applications

Programming in C is fairly well documented and an ODROID “C Tinkering Kit” is sold separately. The wiringPi library is used in C applications.

Below is a C version of the Python example above. It is important to note that these 2 examples talk to the same physical pins but the C wiringPi library uses the ODROID-C2 wPi numbers and the Python RPi.GPIO library uses the Pi BCM numbers.

// led.c - Read/Write "C" example for an Odroid-C2
//
#include  <wiringPi.h>

int main(void)
{
    wiringPiSetup();
    int led = 0;
    int button = 7;
    pinMode(led, OUTPUT);
    pinMode(button, INPUT);
 
    for (;;)
    {
        if (digitalRead(button) == 1) {
           digitalWrite(led, LOW); 
        }
	else
	{
           digitalWrite(led, HIGH); 
        }
    }
    return 0;
}

To compile and run this program:

$ gcc -o led led.c -lwiringPi -lpthread
$ sudo ./led

Summary

I liked that I could reuse some of my Pi cases and prototyping hats with the ODROID-C2.

As a PI user I found that coding in C, Python and NodeRed on the ODROID-C2 was fairly easy, but there were many limitation compared to the Pi platform. The ODROID Wiki had the key product documentation, but it was no where near the incredibly rich documentation that exists with Raspberry Pi modules.

There are a some excellent Python libraries and PI hardware add-ons that support a variety of sensors and I/O applications, these may or not work with the ODROID hardware.

During the development cycle of a project it is nice to have a faster interface, but typically my final projects do not need any high end processing and they can often run on low end Raspberry Pi 1 modules. So for now I would stick to a Raspberry Pi for GPIO/hardware type projects.

I enjoying playing with the ODROID-C2 and for projects requiring higher performance such as video servers, graphic or Web applications then the ODROID-C2 module is definitely worth considering.

IoT with Google Firebase

For many Internet of Things (IoT) projects a message queuing system like MQTT (Message Queue Telemetry Transport) is all that is needed to connect sensors, devices and graphic interfaces together. If however you require a database, with sorting, queuing and multi-media support then there are some great cloud storage platforms that are available. One that is definitely worth taking a look at is Google Firebase.

Like any IoT solution, Google Firebase can have inputs and sensors sending data directly into it, and a variety of client applications to view the data (Figure 1), but Google Firebase also offers other features such as: file storage, machine learning, messaging, and server side functions. In this article I will look at:

In this article I will look at:

  1. Setting up a sample Firebase IoT project
  2. Use Python to simulate I/O data
  3. Create a Web dashboard with Node-Red to view and write data
  4. Create an Android app using AppInventor to view and write data
  5. Look at a more complex data monitoring example in Python

Getting Started

Google Firebase [1] may not have the huge variety of options that Amazon Web Services (AWS) has, but I found as an IoT engineer Google Firebase had all the features that I needed to get up and running quickly, and I didn’t need a credit card for my free activation.

To begin log in with your Google account at https://firebase.google.com/, and select “Get Started”. I created a sample project called My-IoT. After the project is created Firebase will give it a unique identifier name, such as: my-iot-4ded7.

Firebase data can be stored in either a Realtime Database, or a Cloud Firestore. The Realtime Database is an unstructured NoSQL format that has been around for about 4 years, so there are lots of 3rd party components. The Cloud Firestore stores the data in structure collections and it’s fairly new so the 3rd party options are a little more limited.

For our test IoT project we will use the  real-time database, and it can be created from the Database -> Data menu option.

rtdb

The database structure can be built directly from the Firebase web console or imported from a JSON file. For this database 3 grouping are defined, one for Android, Node-Red and Raspberry Pi I/O values. The Firebase console allows for setting, changing and viewing the database values.

db_struct

The security is configured in the Database -> Rules menu option. To keep things simple for testing read and write security is set to true for public access,  (remember to change this for a production system).

security

The project’s remote access settings are shown in the Authentication section with the “Web setup” button.

webconfig

Python and Firebase

There are a number of Python libraries to access Firebase. The pyrebase library has some added functionality such as: queries, sorting, file downloads and streaming support. The pyrebase library only support Python 3, and it is installed by:

sudo pip3 install pyrebase
sudo pip3 install --upgrade google-auth-oauthlib

A Python 3 test program to set two values (pi/pi_value1 and pi/pi_value2) and read a single point and a group of point is shown below. Remember to change the configuration settings to match your project settings.

import pyrebase, random

# define the Firebase as per your settings
config = {
  "apiKey": "AIzaSyCq_WytLdmOy1AIzaSyCq_WytLdmOy1",
  "authDomain": "my-iot-4ded7.firebaseapp.com",
  "databaseURL": "https://my-iot-4ded7.firebaseio.com",
  "storageBucket": "my-iot-4ded7.appspot.com"
}

firebase = pyrebase.initialize_app(config)
db = firebase.database()

# set 2 values with random numbers
db.child("pi").child("pi_value1").set(random.randint(0,100))
db.child("pi").child("pi_value2").set(random.randint(0,100))

# readback a single value
thevalue = db.child("pi").child("pi_value1").get().val()
print ("Pi Value 1: ", thevalue)

# get all android values
all_users = db.child("android").get()
for user in all_users.each():
    print(user.key(), user.val())

The Firebase web console can be used to check the results from the test program.

Node-Red and Firebase

Node-Red is a visual programming environment that allows users to create applications by dragging and dropping nodes on the screen. Logic flows are then created by connecting the different nodes together.

Node-Red has been preinstalled on Raspbian Jesse since the November 2015. Node-Red can also be installed on Windows, Linux and OSX.  To install and run Node-Red on your specific system see https://nodered.org/docs/getting-started/installation.

To install the Firebase components, select the Manage palette option from the right side of the menu bar. Then search for “firebase” and install node-red-contrib-firebase.

nr_fb_install

For our Node-Red example we will use web gauges to show the values of our two Python simulated test points. We will also send a text comment back to our Firebase database. The complete Node-Red logic for this is done in only 6 nodes!

nr_logic

To read the Pi values two firebase.on() nodes are used. The output from these nodes is connected to two dashboard gauge nodes. Double-clicking on firebase_on() node to configure Firebase database and the item to read from. Double-clicking on the gauge node allows you to edit gauge properties.

nr_value1 To send a text string to Firebase a text input node is wired to a firebase modify node. Edit the firebase modify node with the correct database address and value to set.

nr_modify

After the logic is complete, hit the Deploy button on the right side of the menu bar to run the logic.  The Node-Red dashboard user interface is accessed by: http://ipaddress:1880/ui, so for example 192.168.1.108:1880/ui.

nr_screen

AppInventor

AppInventor is a Web based Android app creation tool (http://appinventor.mit.edu), that uses a graphical programming environment.

AppInventor has two main screens. The Designer screen is used for the layout or presentation of the Android app and the Blocks screen is used to build the logic. On the right side of the top menu bar, the Designer and Blocks buttons allow you to toggle between these two screens.

On the Designer screen, an app layout is created by dragging a component from the Palette window onto the Viewer window.

We will use AppInventor to create an Android app that will read Pi values from our Firebase IoT database, and write a value back.

For the visuals on this application a Button, Label (2),  ListView, Textbox and FirebaseDB component will be used. After a component is added to the application, the Components window is used to rename or delete that component. The Properties window is used to edit the features on a component. For the FirebaseDB component configure the Token and URL to match with your project settings.

fb_app_setup

Once the layout design is complete, logic can be added by clicking on the Blocks button on the top menu bar.

Logic is built by selecting an object in the Blocks window, and then click on the specific block that you would like to use.

AppInventor is pretty amazing when it comes to doing some very quick prototyping. Our entire app only uses two main blocks.

The when FirebaseDB1.DataChanged block is executed whenever new data arrives into the Firebase database. The DataChanged block returns a tag and a value variable. The tag variable is the top level item name, (“pi” or “nodered” for our example). The value will be a string of the items and their values, for example: “{pi_value2 = 34, pi_value1=77}”. We use a replace all text block to remove the “{” and “}” characters, then we pass the string to the ListView component. Note this same code will pass 2 or 200 tags in the pi tag section.

The when BT_PUT.click block will pass the text that we enter on the screen into the android/and_value1 item in our database.

fb_app_code

After the screen layout and logic is complete, the menu item Build will compile the app. The app can be made available as an APK downloadable file or as a QR code link.

Below is picture of the final Android app synced with the Firebase data.

Firebase2

Python Data Monitoring Example

Our starting IoT Firebase realtime database was quite simple. Data monitoring or SCADA (Supervisory Control and Data Aquisition) projects usually require more information than a just a value. A more realistic sensor database would includes fields such as tag name, description, status, time and units.

fb_db2

By adding some indexing in the Firebase Security Rules it is possible to create some queries and sorts of the data. Some typical queries would be: “Points in alarm” or “Point values between 2:00 and 2:15”.

fb_rules2

In the code listing below the syntax statement of .order_by_child(“status”).equal_to(“ALARM”).get() is used to show only records that have a status = ALARM. Some other filter options include: .start_at(time1).end_at(time2).get, .limit_to_first(n), and .order_by_value().

A good next step would be to pass important filtered information to Google Firebase’s messaging feature.

import pyrebase, random

# define the Firebase as per your settings
config = {
  "apiKey": "AIzaSyCq_AIzaSyCq_Wyt",
  "authDomain": "my-iot-7ded7.firebaseapp.com",
  "databaseURL": "https://my-iot-7ded7.firebaseio.com",
  "storageBucket": "my-iot-7ded7.appspot.com"
}

firebase = pyrebase.initialize_app(config)
db = firebase.database()

tag_sum = db.child("pi").order_by_child("status").equal_to("ALARM").get()

# print alarm points
for tag in tag_sum.each():
    info = tag.val()
    print (tag.key()," - ", info["description"],":", info["value"],info["units"], info["status"])

For more Python examples see the Pyrebase documentation.

Summary

Without a lot of configuration it is possible to configure a Google Firebase project and have Python, Node-Red and Android app read and write values. The level of programming complexity is on par with an MQTT implementation, however Google Firebase can offer a lot more future functionality that you wouldn’t have with MQTT, such as file storage, machine learning, messaging, and server side functions.

At the time of writing this article I found the Arduino Firebase library to be unreliable but Google offers some options like an MQTT bridge.

 

Pi Sailboat

My daughters and I have built a number of boat projects with an assortment of Arduino, ESP-8266, Bluetooth and RFI components. I believe that this version using a Raspberry Pi and NodeRed offers one of the simplest solutions. This sailboat used a basic catamaran design with a Raspberry Pi mounting inside a waterproof container. Using NodeRed dashboards you can control the sailboat’s rudder from a smart phone. The complete NodeRed logic consisted of only 6 nodes.

Building the Sailboat

There are a lot of different building materials that you could choose from. K’Nex construction pieces are lighter than either Lego or Meccano and they allow you to create reasonably large structures with a minimal number of pieces. If you do not have access to K’Nex pieces then popsicle sticks and some card board would offer a good low cost solution.

To build the sailboat we used:
• K’Nex building pieces
• 4 plastic bottles
• 1 small plastic container with a lid
• String
• Duct tape
• Garbage bag
• Low torque servo
• Raspberry Pi Zero W or Pi 3
• Small USB phone charger

The base of the sailboat was a rectangular structure with 16 down facing K’Nex pieces that allowed plastic bottles to be duct taped in place.

boat_bottom

A few K’Nex pieces were used to create a compartment for the servo, and wire was used to secure the servo in place. A rudder was built by screwing a small piece of wood into the servo arm.

servobox

A garbage bag was cut to the required size and taped to the mast. The boom had a swivel connection to the mast and guide ropes were connected to both the boom and mast.

sailboat_details

Servo and Rudder Setup

Only very low torque servos can connected directly to Rasberry Pi GPIO pins.

Pi_servo_wiring

An example of a low torque servo would be the TowerPro SG90 ($4) that has a torque of 25.00 oz-in (1.80 kg-cm). If you have larger torque servos you will need to either use a custom Raspberry Pi servo hat (there are some good ones on the market), or you will need to use a separate power and ground circuit for the servo.

The wiringPi tool gpio can be used to control the servo. This package is pre-install in the Raspbian image, or it can be manually installed by:

sudo apt-get install -y wiringpi

Servos typically want a pulse frequency of 50 Hz, and the Raspberry Pi PWM (Pulse Width Modulation) pins have a frequency of 19200 Hz, so some range definitions and scaling is required:

gpio -g mode 18 pwm #define pin 18 as the PWM pin
gpio pwm-ms #use 'mark space' mode 
gpio pwmc 192 # set freq as 19200
gpio pwmr 2000 # use a range of 2000

The gpio pwm commands are not persistent after a reboot. A simple solution for this is to put these commands in the Pi user login file of: $HOME/.bash_login.

After the pwm setup commands are run you need to do some manual testing to define your different rudder (servo) positions (Figure 6), such as “Hard Left”, “Hard Right”, “Easy Left”, “Easy Right” and “Straight”. The pwr timing numbers will vary based on your requirements and servo arm positioning, for our sailboat we used:

gpio -g pwm 18 200 #straight
gpio -g pwm 18 260 #hard left
gpio -g pwm 18 140 #hard right
gpio -g pwm 18 230 #easy left
gpio -g pwm 18 170 #easy right

servo_settings

NodeRed Logic and Dashboards

NodeRed is pre-installed on the Raspbian image, but it will need to be set to autostart on a Pi reboot:  sudo systemctl enable nodered.service

NodeRed has a web configuration interface that is accessed by: http://localhost:1880 or http://pi_ip_address:1880.

On the options button (far right), by selecting: View -> Dashboard , you can define and change the web dashboard layouts.

dashboard

To create logic, nodes are selected from the left node panel and dragged and dropped on to the center flow panel. Logic flow are then created by clicking and joining together different inputs and outputs on the nodes. If a dashboard node is dropped on the flow panel it will be added to the default web dashboard. The gpio -g pwm commands can be called using the exec node. The button dashboard node will pass the defined payload value, for example a “Hard Left” 260 is passed when the button is pushed. The button’s payload value will be appended to the exec command to make a complete gpio -g pwm servo position command.

nodered

Once you’ve completed your logic setup press the Deploy button on the top right to make your configuration live and ready to test.

The final step is to enable a smart phone or tablet to connect to the Raspberry Pi, this can be done by either making the Raspberry Pi a WiFi access point or by tethering the Pi to a cell phone. There are some great guides on how to setup a Raspberry Pi as an access point. For this project the simple tethering method was used. Once the Pi is tethered to a phone, the PI’s IP address can be obtained from the hotspot users list.

pi_address

The NodeRed dashboard is accessed on your phone by: http://pi_ip_address:1880/ui .

nodered_ui

Assuming that everything is connected correctly you should be able to control the sailboard with your phone.

Summary

Once you’ve mastered the basic NodeRed and sailboat construction other projects such as motor boats, iceboats, airboats are possible.

airboat

 

 

 

Arduino talking MQTT to Node-Red

There are some great Arduino modules with integrated ESP-8266 wireless chips, some of the more popular modules are:

  • Adafruit HUZZAH
  • NodeMCU
  • WeMos

These modules allow you to do some interesting IoT (Internet of Things) projects. To connect the Arduino modules to PCs, Raspberry Pi’s or Linux nodes that are a number of communication choices. MQTT (Message Queue Telemetry Transport) is becoming one of the standards for this and it is pre-installed with Node-Red.

Plant Moisture Monitoring MQTT Example

Ard_mqtt_overview2

For our example we wanted to do a simple plant moisture example that used a solar charger and an Arduino Wemos module. We then setup an MQTT server on our Node-Red Raspberry Pi with a web dashboard.

Our goal was to get the MQTT technologies working, with some moisture inputs (and not a final plant monitoring system).

Moisture Sensors

Moisture sensors are very low cost and they start at about $2. The basic moisture sensor has 3 inputs; VCC, GND, and AO. Some sensors also include a digital output with a potentiometer to adjust the digital 0-1 moisture limit.

Our Arduino plant moisture setup is good for testing but not a good long term solution. When voltage is applied long term to moisture sensors ionization in the soil will cause a combination of false reading and deterioration of the sensor plates. We plan to do a future project where we will use relays to turn the sensors on/off and we will include solenoid values in a watering system.

MQTT on Arduino

There are a number of excellent MQTT libraries for Arduino, for this example we used the PubSubClient library. This library can be installed from the Arduino IDE by selecting the menu items:

Sketch -> Add Library -> Manage Libraries

To get MQTT setup you’ll need to:

  • define the SSID and password for your WAN
  • define the IP address for the MQTT server (the Node Red/Raspberry Pi node)
  • define some topic for the data

The nice thing about MQTT is that you can define topics for each of your data points. For this example we define the topic humidity to show the moisture sensor value, and msgtext to show the message (‘Needs Water’ or ‘).

Below is our sample Arduino code for passing the moisture data to our MQTT server.

/*
 Basic ESP8266 MQTT publish client example for a moisture sensor
*/
#include <ESP8266WiFi.h>
#include <PubSubClient.h>

// Update these with values suitable for your network.
const char* ssid = "YOUR_SSID_NAME";
const char* password = "YOUR_PASSWORD";
const char* mqtt_server = "YOUR_NODE_RED_IP";

WiFiClient espClient;
PubSubClient client(espClient);

void setup_wifi() {
  // Connecting to a WiFi network
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}

void reconnect() {
  // Loop until we're reconnected
  Serial.println("In reconnect...");
  while (!client.connected()) {
    Serial.print("Attempting MQTT connection...");
    // Attempt to connect
    if (client.connect("Arduino_Moisture")) {
      Serial.println("connected");
    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
      delay(5000);
    }
  }
}

void setup() {
  Serial.begin(9600);
  setup_wifi();
  client.setServer(mqtt_server, 1883);
}

void loop() {
  char msg[10];
  char msgtext[25];
  String themsg;
  if (!client.connected()) {
    reconnect();
  }
  
  int soil_moisture=analogRead(A0);  // read from analog pin A0
  Serial.print("analog value: ");
  Serial.println(soil_moisture);
  
  if((soil_moisture>300)&&(soil_moisture<700)) {
    Serial.println("Humid soil");
    sprintf(msgtext,"Humid soil",soil_moisture);
  } 
  else if ((soil_moisture>700)&&(soil_moisture<950)){
    Serial.println("Moist Soil");
    sprintf(msgtext,"Moist Soil",soil_moisture);
  }
  else if (soil_moisture <300) ){
    Serial.println("Needs water");    
    sprintf(msgtext,"Needs water",soil_moisture);
  }
  else
  {
      sprintf(msgtext,"Sensor Problem",soil_moisture);
  }

  sprintf(msg,"%i",soil_moisture);
  client.publish("humidity", msg);
  client.publish("soil", msgtext);
  delay(5000);
}

Node-Red

Node-Red is an excellent visual programming environment that is part of the Raspberry Pi base install. Node-Red is a simple tool to create your own Internet of Things applications. The base Node-Red installation includes MQTT interfacing components but it does not include an MQTT server.

If you don’t have a Raspberry Pi you can install Node-Red on Window, Mac OS or Linux systems. I’ve had good results running Node-Red on a very old low end laptop running Xubuntu, (see steps 1-3 in this linked guide).

MQTT Server on Node-Red

There are a number of free internet MQTT servers (iot.eclipse.org) that can be used or an MQTT server can be loaded directly on a local server (i.e. Mosquito).

Mosca is a standalone MQTT server that can be installed directly into Node-Red. The Mosca Node-Red component can be either installed at the command line by:

cd $HOME/.node-red

npm install node-red-contrib-mqtt-broker

Or the component can be install within the Node-Red web interface by selecting the “manage palette” option, and then search for mosca.

mosca_install

After Mosca is installed, all that is required is that a “mosca” node needs to be dragged and dropped into the Node-Red project.

mosca_mqtt_nodes

To connect the Arduino module to Node-Red mqtt inputs are added to the project.  The Arduino topics are defined in Node-Red by double-clicking on the mqtt node and then define the topic to match the Arduino topic.

mqtt_topic

After the MQTT connections are configured Web dashboards can present the final data. The Web dashboards offer a number of different components that could be used for this example I used a gauge and a text node.

To compile and view the Node-Red application, click on the Deploy button on the right side of the menu bar. To access the web dashboard enter: http://your-ip:1880/ui . Below is a example of what you should see.

NodeRed_MQTT_GUI

Final Thoughts

The Mosca MQTT server component allows for a simple standalone architecture to connect wireless Arduino modules into a Node-Red IoT solution.

We can also load Node-Red on Raspberry Pi data collection nodes, and have them publish data to a central Node-Red server.

 

 

 

Arduino talking TCP to Node-Red and Python

There are some great Arduino modules with integrated ESP-8266 wireless chips, some of the most popular of these modules are:

  • Adafruit HUZZAH
  • NodeMCU
  • WeMos

Along with these modules comes some excellent libraries.

For Arduino to PC or Raspberry Pi communications that are a few options to choose from. A TCP client/server is simple and straightforward and it is excellent for sending single point information. For sending multiple data points take a look at MQTT (Message Queuing Telemetry Transport), it’s a common standard for IoT applications and it’s built into Node-Red.

Arduino TCP Client

The Arduino module can be a simple TCP client that can talk to either a Python or a Node-Red TCP server. Below is an example that sends a random integer to a TCP server every 5 seconds.

/*
Test TCP client to send a random number
 */
#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>

ESP8266WiFiMulti WiFiMulti;

void setup() {
    Serial.begin(9600);

    // We start by connecting to a WiFi network
    const char * ssid = "your_ssid";       // your WLAN ssid
    const char * password = "your_password"; // your WLAN password
    WiFiMulti.addAP(ssid, password);

    Serial.println();
    Serial.print("Wait for WiFi... ");

    while(WiFiMulti.run() != WL_CONNECTED) {
        Serial.print(".");
        delay(500);
    }
    Serial.println("WiFi connected");
    Serial.println("IP address: ");
    Serial.println(WiFi.localIP());
    delay(500);
}

void loop() {
    const uint16_t port = 8888;          // port to use
    const char * host = "192.168.0.123"; // address of server
    String msg;

    // Use WiFiClient class to create TCP connections
    WiFiClient client;

    if (!client.connect(host, port)) {
        Serial.println("connection failed");
        Serial.println("wait 5 sec...");
        delay(5000);
        return;
    }
   
    // Send a random number to the TCP server
    msg = String(random(0,100));
    client.print(msg);
    Serial.print("Sent : ");
    Serial.println(msg);
    client.stop();    
    delay(5000);
}

Node-Red TCP Server

Node-Red is an excellent visual programming environment that is part of the Raspberry Pi base install. Node-Red is a simple tool to create your own Internet of Things applications. The base Node-Red installation includes a TCP server and client.

To install the web dashboards enter:

sudo apt-get install npm
cd ~/.node-red
npm i node-red-dashboard

To start Node-Red either use the on-screen menus or from the command line enter:

node-red-start &

Once Node-Red is running the programming is done via the web interface at: //localhost:1880 or //your_Pi_ip_address:1880 .

To configure the TCP server, go to the Input nodes section and drag and drop the TCP in node. After the node is inserted double-click on it and edit the port and message settings.

nodered_tcp

To create a gauge Web dashboard, go to the dashboard nodes section and drag and drop the gauge node. After the node is inserted double-click on it and edit the dashboard group, labels and ranges.

nodered_tcp_gauge

For debugging and testing an output debug node is useful.

To access the Web dashboard enter: //localhost:1880/ui or //your_Pi_ip_address:1880/ui

TCP Python Server

The python TCP server will see the incoming Arduino message as a Unicode (UTF-8) text, so convert message to an integer use: thevalue = int(data.decode(“utf-8”)). Below is the full code.

import socket
import sys

HOST = '' # Symbolic name, meaning all available interfaces
PORT = 8888 # Arbitrary non-privileged port

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print ('Socket created')

#Bind socket to local host and port
try:
s.bind((HOST, PORT))
except socket.error as msg:
print ('Bind failed. Error Code : ' + str(msg[0]) + ' Message ' + msg[1])
sys.exit()

print ('Socket bind complete')

#Start listening on socket
s.listen(10)
print ('Socket now listening')

#now keep talking with the client
while True:
#wait to accept a connection - blocking call
conn, addr = s.accept()
data = conn.recv(1024)
print ('Connected with ' + addr[0] + ':' + str(addr[1]) + " " )
thevalue = int(data.decode("utf-8"))
print ("Value: ", thevalue)

s.close(

Summary

In our final application we mounted the Arduino module outside and we powered it with a small solar charger. We also include a humidity value with the temperature, and we used a QR code that linked to our web page.