IoT with Redis Servers

For many Internet of Things (IoT) projects a message queuing system like MQTT (Message Queue Telemetry Transport) is all that is needed to connect sensors, devices and graphic interfaces together. However if you have requirements for high throughput or you’re storing special data types like binary data or image files then you should take a look at Redis.

Redis (Remote Dictorary Server) is an open source, in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets, bitmaps, hyperloglogs and geospatial indexes with radius queries. Redis also supports publish and subscribe methods with wild cards.

In this blog I will show how to set up an Redis system, and then we will look at two examples. The first will be a flame scanner example that connects an Arduino with Redis to a Node-Red dashboard. The second example will be a Raspberry Pi weather station with webcam images that will be shown on a PHP web page.

Getting Started Locally

Redis can be loaded on lower end hardware such as Raspberry Pi’s. To install Redis on Ubuntu system, go to the terminal and type the following commands:

$sudo apt-get update 
$sudo apt-get install redis-server

Once Redis is installed, it can be started by:

$redis-server

The Redis redis-cli command line tools can be used for monitoring and testing. Below is an example of setting and reading a value.

$redis-cli  
127.0.0.1:6379> set mytag1 12.3
OK
127.0.0.1:6379> get mytag1
"12.3"

The default Redis is set to only work only locally. To open it up to remote users edit /etc/redis/redis.conf to comment out all the bind statements, for example #bind 127.0.0.1 . Then restart the service.

Redis Performance

Ok, so I’ve loaded Redis on a Raspberry Pi, a low end Ubuntu laptop and I’ve connected to an Internet hosted system, how can I decide which solution will meet my IoT requirements? Well luckily Redis comes with a tool called redis-benchmark that allows you to do some upfront analysis. The redis-benchmark tool can be configured for different messages, client counts and test durations.

For my IoT project I wanted to check if my my low end Ubuntu laptop could handle 10 clients. My maximum throughput for GET and SET messages would be determined by:

$ redis-benchmark -c 10 -t SET,GET
====== SET ======
100000 requests completed in 18.31 seconds
10 parallel clients
3 bytes payload
...
5460.60 requests per second

====== GET ======
100000 requests completed in 16.96 seconds
10 parallel clients
3 bytes payload
...
5895.53 requests per second

There are many performance factors to consider but for my IoT system with 10 Arduino modules I probably won’t generate more than 50 sets/seconds. According to the bench testing my old Ubuntu laptop appears to be able to do 5,000+ sets/second so I shouldn’t have any problem using it for my IoT system.

Publish and Subscribe

As we’ve seen in the bench marking tool the Redis GET and SET methods are extremely fast. However using a GET to read information requires that you periodically poll to check for new values.

Redis also support a messaging system, where a sender publishes messages, and receivers subscribes to messages. The link that messages are transferred on is called a channel. Unlike a GET command which is a one-time operation the SUBSCRIBE command will automatically send updates on one or a pattern of channels.

Using two redis-cli sessions it is possible to test the publish and subscribe methods. For example a publishing session would be:

 # This is a publishing test session
$ redis-cli
127.0.0.1:6379> PUBLISH mytag2 222
(integer) 0

Then the results in a subscribing session with a wild card would be:

 # This is the subscribing test session
pete@odroidc2:~$ redis-cli
127.0.0.1:6379> PSUBSCRIBE mytag*
Reading messages... (press Ctrl-C to quit)
1) "psubscribe"
2) "mytag*"
3) (integer) 1
1) "pmessage"
2) "mytag*"
3) "mytag1"
4) "111"
1) "pmessage"
2) "mytag*"
3) "mytag2"
4) "222"

Python and Redis

To install the redis-py library use:

pip install redis

A Python example to SET and GET a remote value would be:

# Connect to a local redis server, the set and get a value
import redis
r = redis.Redis(host='localhost', port=6379, password='' ) 
r.set("tag1","123") 
print ("tag1 : ", r.get("tag1"))

A Python example to PUBLISH a value and SUBSCRIBE to a wild card group of values would be:

# Pub/Sub Example
#
import redis, time
r = redis.Redis(host='localhost' )
p = r.pubsub()
r.publish('tag1','5555')
p.subscribe('tag*')

while True:
    message = p.get_message()
    if message:
        if (message['type'] == 'pmessage') :
            print "Channel: ", message['channel'], "=", message['data']
    time.sleep(1)

Flame Scanner Example

For the flame scanner example I wanted to connect a low cost ($3) flame sensor and a temperature sensor to a WiFi enabled Arduino module. The Arduino module would then publish the sensor data to a Redis server. Node-Red subscribes to the Redis data, and presents the information in a web dashboard.

For the flame scanner example I mounted an IR flame sensor on some Meccano pieces and I aimed it at the center of the fireplace. An DHT11 temperature/humidity sensor was also connected to the Arduino module to record the room temperature.

Arduino and Redis

The ESP-8266 family of Arduino modules can be used for sending sensor data to a Redis server. Use the Arduino Library Manager to install the latest Redis version.

Below is a Arduino example that connects to Redis server and publishes three values. The first is the “roomtemp” from the DHT11 temperature sensor, and then the analog “flameintensity” and the digital “flamestatus”. The flame sensor values are normally high, so a digital value of 1=no flame and 0=flame. The same logic holds for the analog value, 0=full flame and 1024=no flame.

#include <Redis.h>
#include "DHT.h"

#define DHTPIN 12 // Signal pin on DHT11
#define DHTTYPE DHT11   // DHT 11 is used
DHT dht(DHTPIN, DHTTYPE);

const int flameAiPin = 13;
const int flameDiPin = 14;

#define WIFI_SSID       "TPLINK"
#define WIFI_PASSWORD   "disco123"

#define REDIS_ADDR      "192.168.0.121" 
#define REDIS_PORT      6379 
#define REDIS_PASSWORD  ""  

Redis redis(REDIS_ADDR, REDIS_PORT);

void setup()
{
    Serial.begin(9600);
    Serial.println();

    WiFi.mode(WIFI_STA);
    WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
    Serial.print("Connecting to the WiFi");
    while (WiFi.status() != WL_CONNECTED)
    {
        delay(250);
        Serial.print(".");
    }
    Serial.println();
    Serial.print("IP Address: ");
    Serial.println(WiFi.localIP());

    if (redis.begin(REDIS_PASSWORD))
    {
        Serial.println("Connected to the Redis server!");
    }
    else
    {
        Serial.println("Failed to connect to the Redis server!");
        return;
    }
}

void loop()
{
  char themsg[10];
  // Get and publish the room temperature
  sprintf(themsg,"%d",int(dht.readTemperature()));
  redis.publish("roomtemp", themsg);

  // Get and publish the flame intensity
  sprintf(themsg,"%d",analogRead(flameAiPin));
  redis.publish("flameintensity", themsg); 

  // Get and publish the flame status
  if (digitalRead(flameDiPin) == 1) {
    redis.publish("flamestatus", "FIRE ON");
  } else {
    redis.publish("flamestatus", "NO FLAME");
  }
  delay(5000);

}

By using the redis-cli psubscribe command you can verify that the Arduino values are being received at the Redis server. An example to subscribe to the roomtemp value would be:

$ redis-cli psubscribe roomtemp
Reading messages... (press Ctrl-C to quit)
...
1) "pmessage"
2) "roomtemp"
3) "roomtemp"
4) "21"

Node-Red and Redis

Node-Red is a visual programming environment that allows users to create applications by dragging and dropping nodes on the screen. Logic flows are then created by connecting the different nodes together.

Node-Red has been preinstalled on Raspbian Jesse since the November 2015. Node-Red can also be installed on Windows, Linux and OSX. To install and run Node-Red on your specific system see https://nodered.org/docs/getting-started/installation.

To install the Redis components, select the Manage palette option from the right side of the menu bar. Then search for “redis” and install node-red-contrib-redis.

For our Node-Red flame scanner example we’ll look at subscribing to 3 items and show them as 2 charts and a text ON/OFF statement on a web dashboard.

The logic for this will require: 3 redis in , 2 chart, and 1 text nodes.

To configure the redis in node, double click on the node and an edit dialog will appear. For this node we will need to configure the redis server, the topic (this is the item to subscribe to, for example “roomtemp” ) and the method is SUBSCRIBE.

The text and gauge nodes are edited by double clicking on them. The key thing here is to define dashboard group. Labels, units and sizing are also configured in this dialog box.

After the logic is complete, hit the Deploy button on the right side of the menu bar to run the logic. The Node-Red dashboard user interface is accessed by: http://ipaddress:1880/ui, so for example 192.168.1.121:1880/ui. Below is a picturing showing the flame scanner web dasboard.

A Weather Station with a Webcam

A Rasp Pi can make an excellent weather station node, not only can sensors such a temperature and humidity be used, but Webcams images can also be included.

For my testing I used low cost USB Webcams but there are also some excellent Pi cameras that can be used. Images can be generated using the fswebcam command line tool, and it is installed by:

sudo apt-get install fswebcam

A Python script to call fswebcam and to upload the image to a Redis server is shown below:

#
# Take a picture and send it to a remote redis server
#
import os
import redis

# have webcam take a picture, save as image1
os.system ('fswebcam --no-banner -r 640x480 image1.jpg')

# enter your redis hostname, port and password
r = redis.Redis(
       host='redis-17303.xxxx.cloud.redislabs.com',
       port=17303, 
       password='zg20s5zIhlPUmZqdtxxxxxx')

img = open("image1.jpg","rb").read()
# save image file into redis, with reference webcam1
r.set("webcam1",img)

For my weather station example I had my Python script periodically read the temperature and humidity and take a picture. This data was SET into a Redis server as “temp1”, “humdity1” and “webcam1”.

I loaded the Redis command line tool (redis-cli) on my PHP/Apache web server and this allowed me to do single line testing that could be moved directly to my PHP script. To pull the image data from Redis I issued a redis-cli command to read raw ‘webcam1’ binary data and write it to a jpeg file. The full PHP file and a sample Web page is shown below.

Final Comments

You have a lot of technologies choices for connecting sensors and devices into your IoT projects. If you’re looking for high speed and for sometime that can support binary data then you should take a look at Redis.