Apache Kafka with Node-Red

Apache Kafka is a distributed streaming and messaging system. There are a number of other excellent messaging systems such as RabbitMQ and MQTT. Where Kafka is being recognized is in the areas of high volume performance, clustering and reliability.

Like RabbitMQ and MQTT, Kafka messaging are defined as topics. Topics can be produced (published) and consumed (subscribed). Where Kafka differs is in the storage of messages. Kafka stores all produced topic messages up until a defined time out.

Node-Red is an open source visual programming tool that connects to Raspberry Pi hardware and it has web dashboards that can be used for Internet of Things presentations.

In this blog I would like to look at using Node-Red with Kafka for Internet of Things type of applications.

Getting Started

Kafka can be loaded on a variety of Unix platforms and Windows.  A Java installation is required for Kafka to run, and it can be installed on an Ubuntu system by:

apt-get install default-jdk

For Kafka downloads and installation instructions see: https://kafka.apache.org/quickstart. Once the software is installed and running there a number of command line utilities in the Kafka bin directory that allow you to do some testing.

To test writing messages to a topic called iot_test1, use the kafka-console-producer.sh  command and enter some data (use Control-C to exit):

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic iot_test1
11
22
33

To read back and listen for messages:

 bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic iot_test1 --from-beginning
11
22
33

The Kafka server is configured in the /config/server.properties  file. A couple of the things that I tweeked in this file were:

# advertised the Kafka server node ip
advertised.listeners=PLAINTEXT://192.168.0.116:9092
# allow topics to be deleted
delete.topic.enable=true

Node-Red

Node-Red is a web browser based visual programming tool, that allows users to create logic by “wiring” node blocks together.  Node-Red has a rich set of add-on components that includes things such as: Raspberry Pi hardware, Web Dash boards, email, Tweeter, SMS etc.

Node-Red has been pre-installed on Raspbian since 2015. For full installation instructions see:  https://nodered.org/#get-started

To add a Node-Red component select the “Palette Manager”, and in the Install tab search for kafka. I found that the node-red-contrib-kafka-manager component to be reliable (but there are others to try).

For my test example I wanted to create a dashboard input that could be adjusted. Then read back the data from the Kafka server and show the result in a gauge.

This logic uses:

  • Kafka Consumer Group – to read a topic(s) from a Kafka server
  • Dashboard Gauge – to show the value
  • Dashboard Slider – allows a user to select a numeric number
  • Kafka Producer – sends a topic message to the Kafka server

nodered_kafka Double-click on the Kafka nodes and in the ‘edit configuration’ dialog create and define a Kafka broker (or server). Also add the topic that you wish to read/write to.

kafka_consume

Double-click on the gauge and slider nodes and define a Dashboard group. Also adjust the labels, range and sizing to meet your requirements.

kafka_gauge

After the logic is complete hit the Deploy button to run the logic. The web dashboard is available at: http://your_node_red_ip:1880/ui.

kafka_phone

Final Comment

I found Node-Red and Kafka to be easy to use in a simple standalone environment. However when I tried to connect to a Cloud based Kafka service (https://www.cloudkarafka.com/) I quickly realized that there is a security component that needs to be defined in Node-Red. Depending on the cloud service that is used some serious testing will probably be required.

 

Raspberry Pi Internet Radio

Node-Red is graphical programming interface that allows logic to be created using nodes that are wired together. Node-Red has been pre-installed in the Raspian OS since 2015 and it has a very easy learning curve.

In this blog I wanted to show an example of Node-Red being used with a five button LCD faceplate to play Internet radio stations.

For this project I used a basic USB powered speaker, a Rasp Pi and a Pi 5-button LCD faceplate. The cost of the faceplates start at about $10.

pi_radio2

Getting Started with Internet Radio

There are a good number of Internet radio resources, https://www.internet-radio.com has a good selection of stations to choose from.

To find a URL of a radio station, look through the stations until you find what you like and then right click on the .pls link, and “Save Link as…”. Save this link as a file and then open the file in a text editor to get the URL.

radio_stations

 

MPD – Music Player Daemon

MPD is a Linux based music service that supports the playing of both music files and internet based radio stations. For command line operations that is also a MPD client application called mpc. To install both the service and client:

sudo apt-get install mpd mpc

Before I started building the node-red application I played with the mpc commands to ensure that I understood the basics.

Internet radio stations are added like a song list:

mpc add 'http://uk2.internet-radio.com:8062'
mpc add 'http://live.leanstream.co/CKNXFM'
mpc add 'http://66.85.88.2:7136'

Some key mpc control commands are:

mpc play  # play the current station
mpc play 3 # play radio station 3
mpc pause  # pause the music
mpc next  # play the next radio station
mpc prev  # play the previous radio station 

mpc volume 90 # set the volume to 90%
mpc volume +5 # increase the volume 5%
mpc volume -5 # decrease the volume 5%

The mpc status command will show the volume, what is playing along with the current station number and total number of stations:

$ mpc status
Comedy104 - A Star104.net Station: Doug Stanhope - To Tell You the Truth
[playing] #2/4 1:45/0:00 (0%)
volume: 75% repeat: off random: off single: off consume: off

Node-Red Logic

Node-Red can be started from the Raspberry Pi menus, or from the command line:

node-red-start &

To access the Node-Red web page, enter the Raspberry Pi ip address with port 1880, for example : http://192.168.0.121:1880

For this project two extra Node-Red components are needed, and they are for the LCD faceplate and the MPD music player. To add components use the “Palette Manager” option.

palette

For the LCD faceplate, search for Adafruit, and select the i2c-lcd-adafruit-sainsmart component.

adafruit_palette

Similarly search for mpd and add the node-red-contrib-mpd component.

mpd_palette To create logic select a node from the left node panel and drag it onto the center flow palette, and then “wire” the nodes together.

For the Internet music example I used four function nodes, and the two i2cLED and the two MPD nodes. (Comment nodes are only used to explain the logic).

node_red_radio

The first step is to double click on the MPD nodes and add an MPD server.

Select Button Logic

I used the select button to turn on and off the music player.

A context variable is created to save the current state of the player. Note: a context variable is only accessible for the node where it is defined..

The ic2_LCD_Input node message has a msg.button_name and msg.button_state item that is used to determine which button is pushed.

For the select button logic a group of messages was used to add the different radio stations.


// create an "is setup" variable
var issetup = context.get('issetup')||0;

if ((msg.button_name == "SELECT") && (msg.button_state == "pressed" )){
// if the setup hasn't been run, add a radio station playlist
if (issetup === 0) {
context.set('issetup',1);
var msg0 = { payload:"clear"};
var msg1 = { payload:"add http://185.33.21.112:11029" }; // 1.FM Trance
var msg2 = { payload:"add http://66.85.88.2:7136" }; // Comedy 104
var msg3 = { payload:"add http://live.leanstream.co/CKNXFM"}; // The One - Wingham
var msg4 = { payload:"add http://185.33.21.112:11269" }; // Baroque
var msg5 = { payload:"play" };
return [ [ msg0, msg1, msg2, msg3, msg4, msg5] ];
} else {
context.set('issetup',0);
msg0 = { payload:"pause"};
return msg0;
}
}

Up/Down Button Logic

The UP button will issue an MPD command equivalent to :

mpc volume +5

This will up the volume by 5%. The total volume will max at 100%.

The DOWN button will issue an MPD command equivalent to :

mpc volume -5

// Raise/Lower the volume
var msg1;
var thevolume = 5; //volume % increment to change

if ((msg.button_name == "UP") && (msg.button_state == "pressed" )){
// if the setup hasn't been run, add a radio station playlist
msg1 = { payload:"volume +" + thevolume };
return msg1;
}
if ((msg.button_name == "DOWN") && (msg.button_state == "pressed" )){
// if the setup hasn't been run, add a radio station playlist
msg1 = { payload:"volume -" + thevolume};
return msg1;
}

Current and Max Radio Station Logic

The ‘Current and Max Radio Stations’ node is updated from the MPD in node when there are any changes to the volume or when a new song or station is played.

This logic creates two flow variables (stnmax, stncnt) that are available in any node in this flow.  The station max (stnmax) and current radio station (stncnt) variables are used in the LEFT/RIGHT button logic to determine which station to change to.


// Get the max number of radio stations and the current radio statio
// Make context variables that can be used in other node, like the LEFT/RIGHT button

var msg1 = msg.payload.status ; //create a simplier message
var stnmax = msg1.playlistlength;
flow.set('stnmax',stnmax);
var stncur = msg1.nextsong;
if (isNaN(stncur)) {stncur = stnmax;} // ensure a valid station

flow.set('stncur',stncur);

return msg1; // only needed for possible debugging

While the code is running it is possible to view the context date.

context_flow

UP/DOWN Button Logic

The UP / DOWN logic changes between the radio stations using the mpc commands:

mpc next
mpc prev

It is important to not move past the range of the radio stations or MPD will hang. The stnmax and stncur variables are used to determine if the next or previous commands are to be allowed.


// Move left and right in radion stations
var stnmax = flow.get('stnmax');
var stncur = flow.get('stncur');
if ((msg.button_name == "LEFT") && (msg.button_state == "pressed" )){
// if the setup hasn't been run, add a radio station playlist
if (stncur > 1) {
var msg0 = {payload:"previous"};
return msg0;
}
}
if ((msg.button_name == "RIGHT") && (msg.button_state == "pressed" )){
// if the setup hasn't been run, add a radio station playlist
if (stncur < stnmax)
var msg1 = {payload:"next"};
return msg1;

}

Final Comments

The Pi LCD faceplate is an excellent hardware add-on for any Raspberry Pi project. However it important to know that clone hardware may work as expected. For my hardware I was not able to easily turn off the extra LED.

A future enhancement would be to add a Web interface so that you could change the volume or stations without using the 5 button Pi faceplate.

 

Pi Network Monitoring

There are some great full featured networking packages like Nagios and MRTG that can be loaded on Raspberry Pi’s. If, however, you are looking for something smaller scale that you can play with then Node-Red might be your answer. Node-Red is a visual programming environment that allows users to create applications by dragging and dropping blocks (nodes) on the screen. Logic flows are then created by connecting wires between the different blocks (nodes). Node-Red also comes with Web Dashboards, so you can view data or do control from your smart phone.

In this blog we’ll look at:

  • running some SNMP commands
  • setup NodeRed for SNMP
  • making read/write SNMP values on the Pi

 

Getting Started with SNMP

Simple Network Management Protocol (SNMP) is the standard for communicating and monitoring of network devices. Common device information is grouped into MIBs or Management Information Bases. Data items are called OIDs or Object Identifiers. OIDs are referenced by either their MIB name or by their OID numeric name. So for example the SNMP device name object could be queried by either its MIB name of: SNMPv2-MIB::sysName.0 or the object identifier number of: .1.3.6.1.2.1.1.5.0.

To install both the SNMP monitor and server on your Pi enter:

sudo apt-get update
sudo apt-get install snmp snmpd snmp-mibs-downloader

To show meaningful MIB names, you will need to modify the SNMP config file by:

sudo nano /etc/snmp/snmp.conf

The first line should be commented out, it should just read: #mibs .

There are many configuration options in the SNMP server agent that need to be considered. For a real/product system you will need to consider your user security but for a test system we can open up the Pi by:

sudo nano /etc/snmp/snmpd.conf

Then uncomment the agentAddress line so that all interfaces are open, and in the Access Control section comment out all the existing user access and add a new line with public access set to read/write (definitely not recommended in a real system):

# Listen for connections on all interfaces (both IPv4 *and* IPv6)
agentAddress udp:161,udp6:[::1]:161

# ACCESS CONTROL
#
# Set read/write access to public anywhere
#
rwcommunity public

After saving the changes to snmpd.conf, the service needs to be restarted:

sudo service snmpd restart

There are a number of useful SNMP command line programs, such as:

  • snmpget – gets a SNMP message for a specific OID
  • snmpset – sets a SNMP OID (OID needs to be writeable)
  • snmpwalk – gets multiple OID values in a MIB tree

The basic syntax for these commands is:

command -c community -v version node OID

To test that SNMP is working, enter the following:

pi@raspberrypi:~ $ snmpwalk -c public -v 1 localhost .1.3 

SNMPv2-MIB::sysDescr.0 = STRING: Linux raspberrypi 4.4.21-v7+ ...
SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.10
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (319234) 0:53:12.34
SNMPv2-MIB::sysContact.0 = STRING: Me <me@example.org>
SNMPv2-MIB::sysName.0 = STRING: raspberrypi
...

If SNMP is working correctly a very long list of SNMP objects will be shown.

Get a Specific SNMP Result

There are thousands of SNMP results that can be read. Some of the more common and useful SNMP are:

1 minute CPU Load: .1.3.6.1.4.1.2021.10.1.3.1
5 minute CPU Load: .1.3.6.1.4.1.2021.10.1.3.2
15 minute CPU Load: .1.3.6.1.4.1.2021.10.1.3.3

Idle CPU time (%): .1.3.6.1.4.1.2021.11.11.0

Total RAM in machine: .1.3.6.1.4.1.2021.4.5.0
Total RAM used: .1.3.6.1.4.1.2021.4.6.0
Total RAM Free: .1.3.6.1.4.1.2021.4.11.0

Total disk/partion size(kBytes): .1.3.6.1.4.1.2021.9.1.6.1
Available space on the disk: .1.3.6.1.4.1.2021.9.1.7.1
Used space on the disk: .1.3.6.1.4.1.2021.9.1.8.1

An example to get the Idle CPU time with the SNMP command line tool would be:

$ snmpget -c public -v 1 localhost .1.3.6.1.4.1.2021.11.11.0

UCD-SNMP-MIB::ssCpuIdle.0 = INTEGER: 97

If the syntax is correct, a result is returned with the OID identifier, result type and the  result value.

Getting Started with NodeRed

NodeRed is pre-installed with the Raspbian images but it will need to have some SNMP and some support options loaded. At a terminal window enter the commands:

sudo apt-get update
sudo apt-get install npm
cd $HOME/.node-red
npm install node-red-node-snmp
npm install node-red-dashboard
npm install node-red-contrib-bigtimer

node-red-start &

Once Node-RED starts, you use a web browser to build applications. If you are working directly on your Pi, enter 127.0.0.1:1880 in the URL address box of your browser. You drop palettes from the left pane into the large flow window in the middle and wire them together in the correct order.

NodeRed Ping Monitor

A good starting program is to make a Web Dashboard that shows ping (node-to-node) delay times. The dashboards are defined in the right panel of Node-Red. A dashboard items are put into groups, and groups are put into tabs. Each tab will be shown as a separate page on your smart phone.

pinglogic

By double-clicking on the Ping node you can enter the different IP address.

pingnode

Similarly by double-clicking on the Chart node, you can define the label and look and feel of the chart.

chartnode

After the configuration is finished, click the Deploy button,  (top right on menu  bar). The Node-RED dashboard user interface is accessed by entering <IPaddress>:1880/ui (e.g., 192.168.1.102:1880/ui). Chart data values are shown by clicking on the chart line.

pingSS

NodeRed with SNMP Nodes

The ping node is quite simple and it returns just the ping value. The snmp node is more complex and it returns multiple pieces of information. To use snmp nodes in a Node-Red program you need some support nodes to parse/pass the payload messages. To send SNMP data to a chart dashboard, the following nodes are used:

  • Big Timer – to trigger the polling of data
  • snmp – gets SNMP/OID information
  • split – split the message into addressable variables
  • change – put the OID value into the message playload
  • chart – show the payload

snmplogic

The SNMP node configuration can get multiple values, a simple example to get the CPU free time is below (Note: the leading “.” is NOT included) :

snmpnode

The split node is used move the SNMP array data in a payload string. The change node is then used to move just the value into the payload.

changenode

The final step is to configure the charts to show the correct label information.

snmpSS

An SNMP Readable Pi Value

The Net-SNMP agent (snmpd) supports the creation of custom read/write objects (OIDs). The “Pass-through” MIB extension command in snmpd.conf allows for script files to be called.

Pass-through script files need to follow a few rules:

  • snmpget request passes a “-g” parameter (get)
  • the snmpget response needs to be 3 lines: OID, data type, and value
  • an snmpset request passes a “-s” parameter (set)

A simple example would be to define the Rasperry Pi board temperature as an SNMP object. The board temp is available by:

cat /sys/class/thermal/thermal_zone0/temp
46160

This returns the value in 1/1000s of a degree C. So to get just the Celsius temperature we can use:

echo $(($(cat /sys/class/thermal/thermal_zone0/temp) / 1000))
46

A SNMP bash script (/home/pi/pitemp), with our Pi CPU temp as OID  .1.3.6.1.4.1.8072.2.1 would be:

#!/bin/bash
if [ "$1" = "-g" ]
then
echo .1.3.6.1.4.1.8072.2.1
echo integer
echo $(($(cat /sys/class/thermal/thermal_zone0/temp) / 1000))
fi
exit 0

After the file is saved remember to make it executable (chmod +x pitemp).

In the “Pass-through” section of  /etc/snmp/snmpd.conf  a line is added to reference the new OID, shell and command:

#
# "Pass-through" MIB extension command
#
pass .1.3.6.1.2.1.8072.2.1 /bin/bash /home/pi/pitemp

After snmpd.conf is updated the snmpd service needs to restarted (sudo service snmpd restart), then our Pi temp OID can be accessed by:

snmpget -c public -v 1 localhost .1.3.6.1.2.1.8072.2.1
NET-SNMP-EXAMPLES-MIB::netSnmpExampleScalars = INTEGER: 47

Pi Writable SNMP GPIO Value

My goal was to use SNMP to turn on and off powered devices, so for this I used a PowerSwitch Tail II, however simple low cost relays could also be used.

The PowerSwitch Tail II ($26) is a power cord that is enabled/disabled with I/O pins. The PowerSwitch pins connect to the Pi pins 6 and 12.

pi_setup

The gpio tool is used to read and write to GPIO pins.  GPIO 1 is made writable by:

gpio mode 1 out

The SNMP script (/home/pi/powerswitch) to read and write to GPIO pin 1 (physical pin 12) is:

#!/bin/bash
if [ "$1" = "-g" ]
then
echo .1.3.6.1.2.1.8072.2.2
echo integer
gpio read 1
fi

if [ "$1" = "-s" ]
then
gpio write 1 $4
fi

exit 0

This new script file needs to made executable by: chmod +x powerswitch. The powerswitch script file is referenced in the SNMP server configuration file (/etc/snmp/snmpd.conf ):

#
# "Pass-through" MIB extension command
#
pass .1.3.6.1.2.1.8072.2.1 /bin/bash /home/pi/pitemp
pass .1.3.6.1.2.1.8072.2.2 /bin/bash /home/pi/powerswitch

Once again the smnpd needs to be restarted. Our read/write actions can be tested by:

$ snmpget -c public -v 1 localhost .1.3.6.1.2.1.8072.2.2
SNMPv2-SMI::mib-2.8072.2.2 = INTEGER: 0
$ snmpset -c public -v 1 localhost .1.3.6.1.2.1.8072.2.2 i 1
SNMPv2-SMI::mib-2.8072.2.2 = INTEGER: 1
$ snmpget -c public -v 1 localhost .1.3.6.1.2.1.8072.2.2
SNMPv2-SMI::mib-2.8072.2.2 = INTEGER: 1

NodeRed Setting SNMP Values

The exec node can be used to call the snmpset command.  A example with an ON and OFF button is:

gpio_logic

The configuration for the exec node is:

exec_node

And the web dashboard is:

gpioSS

Summary

Network monitoring and SNMP is huge topic, hopefully this will give you a good start.

Pi Sailboat

My daughters and I have built a number of boat projects with an assortment of Arduino, ESP-8266, Bluetooth and RFI components. I believe that this version using a Raspberry Pi and NodeRed offers one of the simplest solutions. This sailboat used a basic catamaran design with a Raspberry Pi mounting inside a waterproof container. Using NodeRed dashboards you can control the sailboat’s rudder from a smart phone. The complete NodeRed logic consisted of only 6 nodes.

Building the Sailboat

There are a lot of different building materials that you could choose from. K’Nex construction pieces are lighter than either Lego or Meccano and they allow you to create reasonably large structures with a minimal number of pieces. If you do not have access to K’Nex pieces then popsicle sticks and some card board would offer a good low cost solution.

To build the sailboat we used:
• K’Nex building pieces
• 4 plastic bottles
• 1 small plastic container with a lid
• String
• Duct tape
• Garbage bag
• Low torque servo
• Raspberry Pi Zero W or Pi 3
• Small USB phone charger

The base of the sailboat was a rectangular structure with 16 down facing K’Nex pieces that allowed plastic bottles to be duct taped in place.

boat_bottom

A few K’Nex pieces were used to create a compartment for the servo, and wire was used to secure the servo in place. A rudder was built by screwing a small piece of wood into the servo arm.

servobox

A garbage bag was cut to the required size and taped to the mast. The boom had a swivel connection to the mast and guide ropes were connected to both the boom and mast.

sailboat_details

Servo and Rudder Setup

Only very low torque servos can connected directly to Rasberry Pi GPIO pins.

Pi_servo_wiring

An example of a low torque servo would be the TowerPro SG90 ($4) that has a torque of 25.00 oz-in (1.80 kg-cm). If you have larger torque servos you will need to either use a custom Raspberry Pi servo hat (there are some good ones on the market), or you will need to use a separate power and ground circuit for the servo.

The wiringPi tool gpio can be used to control the servo. This package is pre-install in the Raspbian image, or it can be manually installed by:

sudo apt-get install -y wiringpi

Servos typically want a pulse frequency of 50 Hz, and the Raspberry Pi PWM (Pulse Width Modulation) pins have a frequency of 19200 Hz, so some range definitions and scaling is required:

gpio -g mode 18 pwm #define pin 18 as the PWM pin
gpio pwm-ms #use 'mark space' mode 
gpio pwmc 192 # set freq as 19200
gpio pwmr 2000 # use a range of 2000

The gpio pwm commands are not persistent after a reboot. A simple solution for this is to put these commands in the Pi user login file of: $HOME/.bash_login.

After the pwm setup commands are run you need to do some manual testing to define your different rudder (servo) positions (Figure 6), such as “Hard Left”, “Hard Right”, “Easy Left”, “Easy Right” and “Straight”. The pwr timing numbers will vary based on your requirements and servo arm positioning, for our sailboat we used:

gpio -g pwm 18 200 #straight
gpio -g pwm 18 260 #hard left
gpio -g pwm 18 140 #hard right
gpio -g pwm 18 230 #easy left
gpio -g pwm 18 170 #easy right

servo_settings

NodeRed Logic and Dashboards

NodeRed is pre-installed on the Raspbian image, but it will need to be set to autostart on a Pi reboot:  sudo systemctl enable nodered.service

NodeRed has a web configuration interface that is accessed by: http://localhost:1880 or http://pi_ip_address:1880.

On the options button (far right), by selecting: View -> Dashboard , you can define and change the web dashboard layouts.

dashboard

To create logic, nodes are selected from the left node panel and dragged and dropped on to the center flow panel. Logic flow are then created by clicking and joining together different inputs and outputs on the nodes. If a dashboard node is dropped on the flow panel it will be added to the default web dashboard. The gpio -g pwm commands can be called using the exec node. The button dashboard node will pass the defined payload value, for example a “Hard Left” 260 is passed when the button is pushed. The button’s payload value will be appended to the exec command to make a complete gpio -g pwm servo position command.

nodered

Once you’ve completed your logic setup press the Deploy button on the top right to make your configuration live and ready to test.

The final step is to enable a smart phone or tablet to connect to the Raspberry Pi, this can be done by either making the Raspberry Pi a WiFi access point or by tethering the Pi to a cell phone. There are some great guides on how to setup a Raspberry Pi as an access point. For this project the simple tethering method was used. Once the Pi is tethered to a phone, the PI’s IP address can be obtained from the hotspot users list.

pi_address

The NodeRed dashboard is accessed on your phone by: http://pi_ip_address:1880/ui .

nodered_ui

Assuming that everything is connected correctly you should be able to control the sailboard with your phone.

Summary

Once you’ve mastered the basic NodeRed and sailboat construction other projects such as motor boats, iceboats, airboats are possible.

airboat

 

 

 

Control with Texts (Node-Red on Android)

In places where Internet connections are not possible or too expensive,  SMS text messaging can be a simple approach for monitoring and controlling your remote systems. Many of the mobile providers offer IoT packages for low data throughput, where you’d be looking at spending $1 to $5  per month for 1-5 MB of data.  From the hardware standpoint there are many options such digital modem modules that come in either a Raspberry Pi top or a USB form factor.

If you are looking at doing some prototyping, using an Arduino phone and Node-Red is a great way to get jump started.

Node-Red on Android

Node-Red is a graphical programming system that is used for Internet of Things (IoT) projects. Node-Red is installed in the base Raspberry Pi images, but it can also be loaded on Linux, Windows, MacOS and Android systems.

To load Node-Red on Arduino, you will first need to load Termux,  an Android Terminal Emulator app, that is available at Google Play. After Termux is loaded, enter the following commands to install  and run Node-Red:

apt update
apt upgrade
apt install coreutils nano nodejs
npm i -g --unsafe-perm node-red
node-red

Node-Red will start a Web graphical interface that is accessed by:  http://the_ip_address:1880.

nodered_pc

Extra features can be added and removed from Node-Red by selecting the “Manage Palette” menu option. For this project I needed terminux-api for the texting support, and bigtimer to scanning the text message buffer.

manage_palette

A Test Setup

For a basic setup I used:

  • 1 Android phone with Termux and Node-Red
  • 1 Raspberry Pi with Node-Red
  • 1 Powerswitch Tail II connected to a light
  • 1 Android phone for texting in

sms_pi_setup

Scanning for Text Messages

To create a simple text message application on the Android Note-Red system, the following components are used:

  • A Big Timer node is used to define how often the SMS inbox is scanned. Without doing any configuration, the second output will offer a 1/minute cycle time.
  • An SMS Inbox node will read in a defined number of text messages. To get the last message select the buffer limit to be 1.
  • A function node is used with some JavaScript to check for valid text messages
  • A TCP out node will send a message to another Node-Red system.

A basic logic setup will be as follows:

sms_in_logic

The function node needs some logic to interpret the latest text message and send out the required logic to the Raspberry Pi GPIO pins. The example logic uses a text message of  “LIGHTS ON” or “LIGHTS OFF” to control the output on GPIO pin 12.


// look for new action texts
smslast = context.get('smslast')|| 0;

// Do an action if there is a new SMS message
if ( msg.payload[0].received != smslast) {
    context.set('smslast', msg.payload[0].received )
    smsbody = msg.payload[0].body;
    if (smsbody.toUpperCase().trim() == "LIGHTS ON") {
        msg.payload = 1;  // this is sent to the Pi via TCP
        return msg;
    }
    if (smsbody.toUpperCase().trim() == "LIGHTS OFF") {
        msg.payload = 0;  // this is sent to the Pi via TCP
        return msg;
    }
}

The first time that the code runs you will be prompted on the phone to “Allow Termux:API to send and view SMS messages?”. After you acknowledge this message your Node-Red logic will run cleanly.

Reading TCP Inputs

Android phones can not be directly connected I/O devices, so TCP connections can used to pass commands to Raspberry Pi’s which have the I/O support. Node-Red is pre-installed on most Raspberry Pi images so no added installation is required.

On the Raspberry Pi Node-Red side only a couple of nodes are required:

  • A TCP in node is used to read the incoming communications. For the configuration of this node it is important to have the port (8888 in this case) match the TCP out node’s port. Also the output should be set as: single, and string.
  • An RPI GPIO out node is used set Raspberry Pi General Purpose (GPIO) pins.

sms_tcpin

Texting Out from Android

If your text scheme is simple, for example one light that is turned on and off, then you could manually just output an “ON” or “OFF” message. However if you are controlling multiple lights, heaters and other devices then manually typing the text messages gets awkward.

To manage multiple commands, I wrote a simple Android app in MIT’s App Inventor. App Inventor is a free online Android development environment, that only requires a Google login for access. In about 10 minutes I was able to get a prototype going with multiple inputs. The first step is to drag and drop some buttons from the User Interface Palette onto the viewer pane. Then drag and drop a Texting component from the Social Palette onto the view. Note, the Texting component will appear as a non-visible item.

text2red_gui

After you have a basic layout, select the Blocks button on the menu bar, and this will open a logic page. Logic is created by clicking on the objects in the Block panel, and then dragging and dropping the block onto the Viewer panel.

To have a button sent a text message the following pieces are required:

  • “when Button.Click”  will be called when the users touches the button object
  • “set Texting1.PhoneNumber”  defines the remote phone number
  • “set Texting1.Message” defines the text message
  • “call Texting.SendMessage”  will send the message

text2red_blocks

To build the application use the Build menu item on the menu bar.

app2qr

For my test project I configured four devices and eight commands.text2sms_app

Final Comments

I found that using Node-Red on Android to be a lot faster than I expected, however I noticed that some of the added features (like Bluetooth support) only worked on the Raspberry Pi/Linux version of the Node-Red.

For a final solution I would definitely move to dedicated SMS hardware, but I found it nice to be able to do proof of concept testing with just some basic Android phones. Also don’t forget to setup Node-Red for automatic startup on a power up.

Arduino talking MQTT to Node-Red

There are some great Arduino modules with integrated ESP-8266 wireless chips, some of the more popular modules are:

  • Adafruit HUZZAH
  • NodeMCU
  • WeMos

These modules allow you to do some interesting IoT (Internet of Things) projects. To connect the Arduino modules to PCs, Raspberry Pi’s or Linux nodes that are a number of communication choices. MQTT (Message Queue Telemetry Transport) is becoming one of the standards for this and it is pre-installed with Node-Red.

Plant Moisture Monitoring MQTT Example

Ard_mqtt_overview2

For our example we wanted to do a simple plant moisture example that used a solar charger and an Arduino Wemos module. We then setup an MQTT server on our Node-Red Raspberry Pi with a web dashboard.

Our goal was to get the MQTT technologies working, with some moisture inputs (and not a final plant monitoring system).

Moisture Sensors

Moisture sensors are very low cost and they start at about $2. The basic moisture sensor has 3 inputs; VCC, GND, and AO. Some sensors also include a digital output with a potentiometer to adjust the digital 0-1 moisture limit.

Our Arduino plant moisture setup is good for testing but not a good long term solution. When voltage is applied long term to moisture sensors ionization in the soil will cause a combination of false reading and deterioration of the sensor plates. We plan to do a future project where we will use relays to turn the sensors on/off and we will include solenoid values in a watering system.

MQTT on Arduino

There are a number of excellent MQTT libraries for Arduino, for this example we used the PubSubClient library. This library can be installed from the Arduino IDE by selecting the menu items:

Sketch -> Add Library -> Manage Libraries

To get MQTT setup you’ll need to:

  • define the SSID and password for your WAN
  • define the IP address for the MQTT server (the Node Red/Raspberry Pi node)
  • define some topic for the data

The nice thing about MQTT is that you can define topics for each of your data points. For this example we define the topic humidity to show the moisture sensor value, and msgtext to show the message (‘Needs Water’ or ‘).

Below is our sample Arduino code for passing the moisture data to our MQTT server.

/*
 Basic ESP8266 MQTT publish client example for a moisture sensor
*/
#include <ESP8266WiFi.h>
#include <PubSubClient.h>

// Update these with values suitable for your network.
const char* ssid = "YOUR_SSID_NAME";
const char* password = "YOUR_PASSWORD";
const char* mqtt_server = "YOUR_NODE_RED_IP";

WiFiClient espClient;
PubSubClient client(espClient);

void setup_wifi() {
  // Connecting to a WiFi network
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}

void reconnect() {
  // Loop until we're reconnected
  Serial.println("In reconnect...");
  while (!client.connected()) {
    Serial.print("Attempting MQTT connection...");
    // Attempt to connect
    if (client.connect("Arduino_Moisture")) {
      Serial.println("connected");
    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
      delay(5000);
    }
  }
}

void setup() {
  Serial.begin(9600);
  setup_wifi();
  client.setServer(mqtt_server, 1883);
}

void loop() {
  char msg[10];
  char msgtext[25];
  String themsg;
  if (!client.connected()) {
    reconnect();
  }
  
  int soil_moisture=analogRead(A0);  // read from analog pin A0
  Serial.print("analog value: ");
  Serial.println(soil_moisture);
  
  if((soil_moisture>300)&&(soil_moisture<700)) {
    Serial.println("Humid soil");
    sprintf(msgtext,"Humid soil",soil_moisture);
  } 
  else if ((soil_moisture>700)&&(soil_moisture<950)){
    Serial.println("Moist Soil");
    sprintf(msgtext,"Moist Soil",soil_moisture);
  }
  else if (soil_moisture <300) ){
    Serial.println("Needs water");    
    sprintf(msgtext,"Needs water",soil_moisture);
  }
  else
  {
      sprintf(msgtext,"Sensor Problem",soil_moisture);
  }

  sprintf(msg,"%i",soil_moisture);
  client.publish("humidity", msg);
  client.publish("soil", msgtext);
  delay(5000);
}

Node-Red

Node-Red is an excellent visual programming environment that is part of the Raspberry Pi base install. Node-Red is a simple tool to create your own Internet of Things applications. The base Node-Red installation includes MQTT interfacing components but it does not include an MQTT server.

If you don’t have a Raspberry Pi you can install Node-Red on Window, Mac OS or Linux systems. I’ve had good results running Node-Red on a very old low end laptop running Xubuntu, (see steps 1-3 in this linked guide).

MQTT Server on Node-Red

There are a number of free internet MQTT servers (iot.eclipse.org) that can be used or an MQTT server can be loaded directly on a local server (i.e. Mosquito).

Mosca is a standalone MQTT server that can be installed directly into Node-Red. The Mosca Node-Red component can be either installed at the command line by:

cd $HOME/.node-red

npm install node-red-contrib-mqtt-broker

Or the component can be install within the Node-Red web interface by selecting the “manage palette” option, and then search for mosca.

mosca_install

After Mosca is installed, all that is required is that a “mosca” node needs to be dragged and dropped into the Node-Red project.

mosca_mqtt_nodes

To connect the Arduino module to Node-Red mqtt inputs are added to the project.  The Arduino topics are defined in Node-Red by double-clicking on the mqtt node and then define the topic to match the Arduino topic.

mqtt_topic

After the MQTT connections are configured Web dashboards can present the final data. The Web dashboards offer a number of different components that could be used for this example I used a gauge and a text node.

To compile and view the Node-Red application, click on the Deploy button on the right side of the menu bar. To access the web dashboard enter: http://your-ip:1880/ui . Below is a example of what you should see.

NodeRed_MQTT_GUI

Final Thoughts

The Mosca MQTT server component allows for a simple standalone architecture to connect wireless Arduino modules into a Node-Red IoT solution.

We can also load Node-Red on Raspberry Pi data collection nodes, and have them publish data to a central Node-Red server.

 

 

 

Arduino talking TCP to Node-Red and Python

There are some great Arduino modules with integrated ESP-8266 wireless chips, some of the most popular of these modules are:

  • Adafruit HUZZAH
  • NodeMCU
  • WeMos

Along with these modules comes some excellent libraries.

For Arduino to PC or Raspberry Pi communications that are a few options to choose from. A TCP client/server is simple and straightforward and it is excellent for sending single point information. For sending multiple data points take a look at MQTT (Message Queuing Telemetry Transport), it’s a common standard for IoT applications and it’s built into Node-Red.

Arduino TCP Client

The Arduino module can be a simple TCP client that can talk to either a Python or a Node-Red TCP server. Below is an example that sends a random integer to a TCP server every 5 seconds.

/*
Test TCP client to send a random number
 */
#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>

ESP8266WiFiMulti WiFiMulti;

void setup() {
    Serial.begin(9600);

    // We start by connecting to a WiFi network
    const char * ssid = "your_ssid";       // your WLAN ssid
    const char * password = "your_password"; // your WLAN password
    WiFiMulti.addAP(ssid, password);

    Serial.println();
    Serial.print("Wait for WiFi... ");

    while(WiFiMulti.run() != WL_CONNECTED) {
        Serial.print(".");
        delay(500);
    }
    Serial.println("WiFi connected");
    Serial.println("IP address: ");
    Serial.println(WiFi.localIP());
    delay(500);
}

void loop() {
    const uint16_t port = 8888;          // port to use
    const char * host = "192.168.0.123"; // address of server
    String msg;

    // Use WiFiClient class to create TCP connections
    WiFiClient client;

    if (!client.connect(host, port)) {
        Serial.println("connection failed");
        Serial.println("wait 5 sec...");
        delay(5000);
        return;
    }
   
    // Send a random number to the TCP server
    msg = String(random(0,100));
    client.print(msg);
    Serial.print("Sent : ");
    Serial.println(msg);
    client.stop();    
    delay(5000);
}

Node-Red TCP Server

Node-Red is an excellent visual programming environment that is part of the Raspberry Pi base install. Node-Red is a simple tool to create your own Internet of Things applications. The base Node-Red installation includes a TCP server and client.

To install the web dashboards enter:

sudo apt-get install npm
cd ~/.node-red
npm i node-red-dashboard

To start Node-Red either use the on-screen menus or from the command line enter:

node-red-start &

Once Node-Red is running the programming is done via the web interface at: //localhost:1880 or //your_Pi_ip_address:1880 .

To configure the TCP server, go to the Input nodes section and drag and drop the TCP in node. After the node is inserted double-click on it and edit the port and message settings.

nodered_tcp

To create a gauge Web dashboard, go to the dashboard nodes section and drag and drop the gauge node. After the node is inserted double-click on it and edit the dashboard group, labels and ranges.

nodered_tcp_gauge

For debugging and testing an output debug node is useful.

To access the Web dashboard enter: //localhost:1880/ui or //your_Pi_ip_address:1880/ui

TCP Python Server

The python TCP server will see the incoming Arduino message as a Unicode (UTF-8) text, so convert message to an integer use: thevalue = int(data.decode(“utf-8”)). Below is the full code.

import socket
import sys

HOST = '' # Symbolic name, meaning all available interfaces
PORT = 8888 # Arbitrary non-privileged port

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print ('Socket created')

#Bind socket to local host and port
try:
s.bind((HOST, PORT))
except socket.error as msg:
print ('Bind failed. Error Code : ' + str(msg[0]) + ' Message ' + msg[1])
sys.exit()

print ('Socket bind complete')

#Start listening on socket
s.listen(10)
print ('Socket now listening')

#now keep talking with the client
while True:
#wait to accept a connection - blocking call
conn, addr = s.accept()
data = conn.recv(1024)
print ('Connected with ' + addr[0] + ':' + str(addr[1]) + " " )
thevalue = int(data.decode("utf-8"))
print ("Value: ", thevalue)

s.close(

Summary

In our final application we mounted the Arduino module outside and we powered it with a small solar charger. We also include a humidity value with the temperature, and we used a QR code that linked to our web page.