DynamoDB with Python and JavaScript

Amazon Web Services (AWS) is a probably the best known Cloud services offering on the Internet.  At first glance AWS is a quite intimidating, learning about all the different features of AWS could be a full time job. For this blog I wanted to introduce one component of AWS called DynamoDB.

DynamoDB is a NoSQL database that could be connected to sensor hub devices such as Raspberry Pi’s for Internet of Things (IoT) solutions.

aws_services

Getting Started with DynamoDB

The first step to getting started with AWS is to create an account. You can open a free “testing” account but unfortunately you’ll need to show that you’re serious by including a credit card number.

After creating an account you can sign into the AWS Management Console (https://aws.amazon.com/console/). With a single user account it is possible to run different services in different locations. It is important to note that not all services are available in all locations, so select a location that offers all the services that you need.

AWS_Region

After a region is selected go to the Services menu item and search for DynamoDB, next select “create table”. A DynamoDB table has a NoSQL database structure, so this means that all that is required is a table name and a primary key. A primary key allows rows of data to have a way to be connected together.

For my IoT table example I created a primary key called signalname. (If you plan to have a very large data set and you need extremely high speed response you might want to consider defining multiple keys, such as location or sensor type).

create_table

Once the table is created, you can view and modify data.

the_table

The “Items” tab is used to add, view and delete data. Note, the viewing of updates is not automatic, so you need to press the “refresh” button periodically.

mytag1

Users and Security

The AWS user security is quite detailed. To get started select your account name on the menu bar and “My Security Credentials“.  The Groups item allows you to create security classifications for activities such as admin, viewing, etc.

New users can be defined and added into security groups.

user_security

When a new users is created an Access Key ID and a Secret Access Key is generated. The Secret Access Key is only shown when the user is created so ensure that you save it, and/or download the CSV file.

newuser

After an Access key ID and Secret Access Key is created for a user, you can start  programming interfaces to DynamoDB.

DynamoDB and Python

The Python AWS library is called boto3 and it is installed  in Windows by :

pip install boto3

and in Linux by:

sudo pip install boto3

The AWS security can be either added directly into the Python (or JavaScript) application or it can reside on the node or PC that you are working on. Both methods have pro’s and con’s. To add the credentials locally, you’ll need to install the AWS command line tool (awscli) and then enter your specific information:

$ sudo pip install awscli
$ aws configure
AWS Access Key ID [None]: AKIAIDxxxxx
AWS Secret Access Key [None]: kzzeko8Fxxxxxxxxxxxxxxxxx
Default region name [None]: Central
Default output format [None]:

A Python example to write and read a row into DynamoDB with the credentials in the program could be:

import boto3
import time

thesec = int(time.time() % 60)

mysession = boto3.session.Session(
    aws_access_key_id= 'AKIAID7xxxxxxxxx',
    aws_secret_access_key='kzzeko8xxxxxxxxxxxxxxxx')
db = mysession.resource('dynamodb', region_name='ca-central-1')

table = db.Table('my_iot_burl1')

response = table.put_item(
   Item={
        'signal': 'mytag1',
        'tagdesc': 'test tag1',
        'tagvalue': thesec
    })

response = table.get_item(Key={'signal': 'mytag1'})
print(thesec)
print(response)
print(response['Item']['tagvalue'])

If the credentials are locally used then the code to write data would be:

import boto3

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('users01')

table.put_item(
   Item={
        'signal': 'mytag1',
        'tagdesc': 'test tag1',
        'tagvalue': 16
    }
)

The response comes back as JSON data. For this example the result would be:

16
{‘Item’: {‘tagvalue’: Decimal(’16’), ‘tagdesc’: ‘test tag1’, ‘signal’: ‘mytag1’}, ‘ResponseMetadata’: {‘RequestId’: ’66M7R4CVCACA4KNA24LM90KG0NVV4KQNSO5AEMVJF66Q9ASUAAJG’, ‘HTTPStatusCode’: 200, ‘HTTPHeaders’: {‘server’: ‘Server’, ‘date’: ‘Fri, 15 Feb 2019 16:29:48 GMT’, ‘content-type’: ‘application/x-amz-json-1.0’, ‘content-length’: ’84’, ‘connection’: ‘keep-alive’, ‘x-amzn-requestid’: ’66M7R4CVCACA4KNA24LM90KG0NVV4KQNSO5AEMVJF66Q9ASUAAJG’, ‘x-amz-crc32’: ‘220267538’}, ‘RetryAttempts’: 0}}
16

For large amounts of data being written to DynamoDB it is possible to use a batch writing function:

import boto3
import time

db = boto3.resource('dynamodb', region_name='ca-central-1')

table = db.Table('my_iot_burl1')

thesec = int(time.time() % 60)

with table.batch_writer() as batch:
    for i in range(1,11):
        batch.put_item(
            Item={
                'signal': 'mytag' + str(i),
                'tagdesc': 'test tag'+ str(i),
                'tagvalue': thesec,
                'status': 'NORMAL'
            }
        )

The AWS console can be used to verify that the data was written in.

Writing to DynamoDB will clear the existing row with a completely new set of data. If you only want to update a field in a row then the update_item call is used. In the example below only the sensor tagvalue is updated and all other fields are left unchanged.

import boto3

mysession = boto3.session.Session(
    aws_access_key_id= 'AKIAIDxxxxxxxxx',
    aws_secret_access_key='kzzekxxxxxxxxxxxxxxx'
)

db = mysession.resource('dynamodb', region_name='ca-central-1')

table = db.Table('my_iot_burl1')

# set the value for mytag1 to bet 29
table.update_item(
    Key={ 'signal': 'mytag1' },
    UpdateExpression='SET tagvalue = :newval1',
    ExpressionAttributeValues={
        ':newval1': 29
    }
)

Python “Scan the Data” Example

The boto3 library supports the ability to scan the data based on some filtering criteria. The filter syntax is: Attr(‘the field’).eq(‘thevalue’).

The following comparison operators are available:

eq | ne | le | lt | ge | gt | not_null | null | contains | not_contains | begins_with | in | between

Multiple statements can be created with | (OR) , & (AND) operators. Below is an example that scans the database for rows with an alarm status or with a tag value greater than 100.

scan_view

import boto3
from boto3.dynamodb.conditions import Key, Attr

db = boto3.resource('dynamodb', region_name='ca-central-1')

table = db.Table('my_iot_burl1')

response = table.scan(
    FilterExpression=Attr('status').eq('ALARM') | Attr('tagvalue').gt(100)
)

#print (response)
for i in response['Items']:
    print(i)
    print(i['signal'], "=", i['tagvalue'])

The results were:

{‘tagvalue’: Decimal(‘150’), ‘signal’: ‘mytag2’, ‘tagdesc’: ‘test tag2’, ‘status’: ‘NORMAL’}
mytag2 = 150
{‘tagvalue’: Decimal(‘999’), ‘signal’: ‘mytag1’, ‘tagdesc’: ‘test tag1’, ‘status’: ‘ALARM‘}
mytag1 = 999

JavaScript

For standard HTML/JavaScript projects the credentials are included in the source file. This obviously has some serious security issues, so it’s recommended that you define a low level user that can only do a small subset of functions.

Below is a simple example that can read our DynamoDB table and it can write a new row with an updated value.

<html>
<head>
<script src="https://sdk.amazonaws.com/js/aws-sdk-2.7.16.min.js"></script>

<script>
AWS.config.update({
  region: "ca-central-1",
  accessKeyId: "AKIAID7xxxxxxxxxxxx",
  secretAccessKey: "kzzeko8xxxxxxxxxxxxx"
});

var docClient = new AWS.DynamoDB.DocumentClient();

function createItem() {
  var d = new Date();
  var n = d.getSeconds();
    var params = {
        TableName :"my_iot_burl1",
        Item:{
      'signal': 'mytag4',
      'desc': 'test tag4',
      'info': {
        'value':n,
        'quality': "questionable"
        }
    }
    };
    docClient.put(params, function(err, data) {
        if (err) {
            document.getElementById('textarea').innerHTML = "Unable to add item: " + "\n" + JSON.stringify(err, undefined, 2);
        } else {
            document.getElementById('textarea').innerHTML = "PutItem succeeded: " + "\n" + JSON.stringify(params,undefined, 2);
        }
    });
}

function readItem() {
    var table = "my_iot_burl1";
    var signal = 'mytag4';
    var params = {
        TableName: table,
        Key:{'signal' : 'mytag4' }
    };
    docClient.get(params, function(err, data) {
        if (err) {
            document.getElementById('textarea2').innerHTML = "Unable to read item: " + "\n" + JSON.stringify(err, undefined, 2);
        } else {
            document.getElementById('textarea2').innerHTML = "GetItem succeeded: " + "\n" + JSON.stringify(data, undefined, 2);
        }
    });
}

</script>
</head>

<body>
<input id="createItem" type="button" value="Create Item" onclick="createItem();" />
<br><br>
<textarea readonly id= "textarea" style="width:400px; height:300px"></textarea>
<hr>
<input id="readItem" type="button" value="Read Item" onclick="readItem();" />
<br><br>
<textarea readonly id= "textarea2" style="width:400px; height:300px"></textarea>
</body>
</html>

aws_html

Summary

On first impressions I was pretty overwhelmed using Amazon Web Services. However after some time playing with it I found that it wasn’t unlike many other Web services that are available.

DynamoDB can be used to communicate with sensor values from a Raspberry Pi, but unfortunately there are no mainstream libraries for Arduino projects. AWS does offer an MQTT solution that can be used to update a DynamoDB table.

For small IoT projects I feel that AWS is overkill, but for larger projects I would definitely consider using it.

MicroPython Air Boat

My daughter’s and I have built a number of air boat projects, but this time I thought that I’d try it using MicroPython. MicroPython is a lean and efficient implementation of the Python 3 programming language that includes a small subset of the Python standard library and is optimised to run on microcontrollers.

For this project my goal was to create a MicroPython application that was a standalone WiFi access point, and its used a small web server for controlling the air boat fans.

Hardware

The hardware that I used on this project included:

• 1 – ESP8266 module ($5-$10)
• 2 – L9110 Fans ($5 each)
• 1 – Uno proto shield (optional) or use a small breadboard
• 9V battery or portable phone charger
• K-Nex blocks
• small plastic box
• 2 – plastic water bottles
• duct tape

MicroPython is supported on a number of Wifi enabled ESP32 and ESP8266 modules. These modules are well priced with the NodeMCU modules starting around $5. It’s important to note that most of the ESP-8266 modules also support Lua and Arduino C/C++ programming.

nodemcu

For this project I used an Wemos ESP-8266 module, that comes in a Arduino Uno form factor. I like the Wemos modules because I can used my Uno proto shields and these modules supports 5V DC, as opposed to the typical MicroPython modules that only support 3.3V. The fans will work with 3.3V but they generate a lot more wind power at 5V.

The L9110 fans are designed for Arduino projects and they only cost about $5. These fans have four pins: VCC, GND, INA (direction), and INB (on/off). For this project I only used forward spinning fans, so only pin INB was used.

L9110_fan

Boat Construction

For the boat frame I used K’Nex pieces because they are light weight and sturdy, however there a lot of other materials that could be used. Water bottles were used for flotation, and duct tape was used to secure the bottles to the frame. To help protect the electronics a small plastic snack container was used. I wire wrapped the fans to the boat frame.

airboat

MicroPython Setup

There are a few choices for MicroPython development. I found that the uPyCraft IDE offered a nice integrated environment (Figure 5), and there are some excellent tutorials to help you get started.

upycraft

For MicroPython projects you typically create 2 Python applications, a boot.py and a main.py file. I like to equate this to Arduino where you have a setup() function and a loop() function. For this example I wanted to keep the documentation simple so I put everything in the boot.py file, but it’s recommended to use main.py for larger projects.

Creating an Access Point

The picture below shows how to setup an access point in just a few lines of code. For this example the access point is called ‘ESP32’ with a password of ‘12345678’. When the code is run you will be able to see when a remote user connects and disconnects to the access point.

access_point

MicroPython Web Server

After getting the Access Point working the next step is to create a Web server. This is a simple Web server project so I embedded the HTML content into my Python code, however for a more complex application I would definitely have my web pages as files that are independent from the code.

For the Web Server example application , we start with the access point connection code, then we setup a socket on port 80. The HTTP request/response sequence is passed through a function called web_page(request). In web_page(request) the code looks for keywords in the HREF request.

# MicroPython boot.py - Access Point Web Server

try:
  import usocket as socket
except:
  import socket

import network

station = network.WLAN(network.AP_IF)
station.active(True)
station.config(essid='ESP32')
station.config(authmode=3,password='12345678')

while station.isconnected() == False:
  pass

print('Connection successful')
print(station.ifconfig())

def web_page(request):
  
  fans_state = ""
  if request.find('/?forward') > 0:
    fans_state="Going Forward"
  if request.find('/?Stopped') > 0:
    fans_state="Stopped" 
  
  html = """<html><head> <title>Ice Boat Web Server</title> 
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="icon" href="data:,"> <style>
  html{font-family: Helvetica; display:inline-block; margin: 0px auto; text-align: center;}
  h1{color: #0F3376; padding: 2vh;}p{font-size: 1.5rem;}
  .button{display: inline-block; background-color: #e7bd3b; border: none; 
  border-radius: 4px; color: white; text-decoration: none; font-size: 30px; width:100%}
  </style></head>
  <body> <h1>Ice Boat Web Server</h1> 
  <p>ICE Boat : <strong>""" + fans_state + """</strong></p>
  <p><a href='/?forward'><button class="button">Forward</button></a></p>
  <p><a href='/?stop'><button class="button button">STOP</button></a></p>
  
  </body></html>"""
  
  return html

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('', 80))
s.listen(5)

while True:
  conn, addr = s.accept()
  print('Got a connection from %s' % str(addr))
  request = conn.recv(1024)
  request = str(request)
  print('The Content = %s' % request)
  response = web_page(request)
  conn.send(response)
  conn.close()

The embedded HTML code passes keywords using anchor tags, for example: <a href=’/?forward’>. Often we use mobile frameworks like Bootstrap to help with formatting, however because we’re running a standalone access point we need to manually define all our style codes.

To access our MicroPython web server, use the address: 192.168.4.1. This is the default address (in the access point setup you can change this). When our web server is running we should be able to toggle the stop/forward states.

screen2

Writing Outputs

The MicroPython command line interface is good way to test outputs. To access the command line interface, use the connect button and then enter “Control-C”.

To manage pins on the hardware, machine library is used :

from machine import Pin

Then pin objects can be defined as either inputs or outputs:

Pin14 = Pin(14, Pin.IN)
Pin5 = Pin(5, Pin.OUT)

The value of a pin is read using: pinobject.value(), and set with: pinobject.value(thevalue) .

motor_test

Final Application

Now that we have all the pieces working independently we can pull it all together. For the final code I’ve defined two fans (motorR and motorL), and a fan control function is called from the web requests.

# MicroPython boot.py - Access Point and Airboat Web Controls
import time
try:
  import usocket as socket
except:
  import socket

from machine import Pin
import network

# Define the left, right and back fans pin
motorR = Pin(12,Pin.OUT)
motorL = Pin(4,Pin.OUT)

# start with fans stopped, Note: my FANs are 0=run
motorR.value(1)
motorL.value(1)


station = network.WLAN(network.AP_IF)
station.active(True)
station.config(essid='ESP32')
station.config(authmode=3,password='12345678')

while station.isconnected() == False:
  pass

print('Connection successful')
print(station.ifconfig())

def fancontrol(left,right):
  motorL.value(left)
  motorR.value(right)

def web_page(request):
  
  fans_state = "Stopped"
  if request.find('/?forward') > 0:
    fans_state="Going Forward"
    fancontrol(0,0)
  if request.find('/?left') > 0:
    fans_state="Going Left"
    fancontrol(1,0)
  if request.find('/?right') > 0:
    fans_state="Going Right" 
    fancontrol(0,1)
  if request.find('/?stop') > 0:
    fans_state="Stopped"
    fancontrol(1,1)
  
  html = """<html><head> <title>Air Boat Web Server</title> 
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="icon" href="data:,"> <style>
  html{font-family: Helvetica; display:inline-block; margin: 0px auto; text-align: center;}
  h1{color: #0F3376; padding: 2vh;}p{font-size: 1.5rem;}
  .button{display: inline-block; background-color: #e7bd3b; border: none; 
  border-radius: 4px; color: white; text-decoration: none; font-size: 30px; width:100%}
  .button2{background-color: #4286f4; width:49%}
  </style></head>
  <body> <h1>Air Boat Web Server</h1> 
  <p>Air Boat : <strong>""" + fans_state + """</strong></p>
  <p><a href='/?forward'><button class="button">Forward</button></a></p>
  <p><a href='/?left'><button class="button button2">LEFT</button></a>
  <a href='/?right'><button class="button button2" >RIGHT</button></a></p>
  <p><a href='/?stop'><button class="button button">STOP</button></a></p>
  
  </body></html>"""
  
  return html

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('', 80))
s.listen(5)

while True:
  conn, addr = s.accept()
  print('Got a connection from %s' % str(addr))
  request = conn.recv(1024)
  request = str(request)
  print('The Content = %s' % request)
  response = web_page(request)
  conn.send(response)
  conn.close()

airboat_moving

Summary

You’ll be quite surprised about how fast even 2 fans will move the air boat. Balancing the direction of the fans or adding a simple rudder may be required to ensure that it keeps a straight forward direction.

I’ve done the same project on the exact same hardware in Anduino C++ and I would say the response speed is similar but the Python code might be slightly leaner. I found that the MicroPython IDE wasn’t as robust as the Arduino IDE, but I really enjoyed doing interactive Python testing from a command prompt.

I won’t be giving up on Arduino C++, but I can definitely see a place for MicroPython, especially for projects with lots of string manipulation.

 

X Windows from Raspberry Pi to Android

So you’ve done a cool graphic app on your Raspberry Pi wouldn’t it be nice to see it on your Android phone ? Well luckily there are a couple of X Server apps that you can run on your Android phone that allows you to see your Pi graphic apps.

X Server Apps for Android

When you do a search for “x servers” on Android Play Store, you will get a few results.

apps_xservers

X Server XSDL from pelya, has good reviews and I found it to be fairly robust. My only comment was I wasn’t sure how to clear the background help statements, and it made the screen look a little messy.

X server from Darkside is a beta release. I preferred the background and SSH integration over XSDL, however this app did not always close the window when the code requested it.

Both options are free and open source.

Getting Started with X Windows

There are two main ways for X Windows to be run on a remote system. The first method is that the Window is sent from the Pi (or any Linux/X Windows system) to the Android phone. The second method is what you typically do on a Windows PC using an SSH terminal program like  Putty; you SSH (Secure Shell) in to the Rasp Pi and then when you run an app it open on your remote node.

xwindow_overview

Calling windows from a PC is pretty easy but calling windows from an Android phone is challenging and slow because of the keyboard. The two X Servers apps do not have SSH components built-in but there are some Android SSH client apps that can be run independently or integrated with the Darkside X app.

Sending an X Window from a Raspberry Pi or Linux node is rather straightforward. The sending location of an X window is defined by the DISPLAY session variable. To check its value enter:

~$ echo $DISPLAY
localhost:10.0

A typical results will be something like: localhost:10.0 .

To change the DISPLAY variable enter the ipaddress:window_number.  In my case the phone’s IP is 192.168.0.102, and the window number is 0, so in a terminal on the PI the command is:

export DISPLAY=192.168.0.102:0

This will now direct all X Windows in my terminal session to my phone. To test things we can send the Python IDE to the phone by:

$ idle

x_idle

Unfortunately the Android X Server app does not support moving or resizing of X Windows, so the app is pinned to the top left. Luckily if we are writing Python apps we can set the sizing and positioning when the app starts up.

Python Tkinter

The tkinter graphic library allows you to create a simple graphic front end to your Python projects.

The following code will update a label with the time every second. The geometry setting is used to move the window (300 left, and 100 down).

# test1.py - Show the time every second
#
import tkinter as tk
from datetime import datetime

root = tk.Tk()
root.geometry("+300+100")

def Updatetime():
label['text'] = datetime.now().strftime("%H:%M:%S")
root.after(1000,Updatetime)

def CallBack():
root.destroy()

label = tk.Label( root, text='', font = "Aria 72 bold" )
label.pack()

B = tk.Button(root, text ="Quit", font = "Aria 72 bold", command = CallBack)

B.pack()

root.after(1000,Updatetime)
root.mainloop()

To set the X window to my phone and run this Python application enter:

~$ export DISPLAY=192.168.0.102:0
~$ python test1.py

test1

Final Comment

I found that Python 3 was much more solid than Python 2. For example a Pi real time bars example worked perfectly on Python3, but on Python2 the bar wouldn’t update at all.

x_bars

 

 

 

 

 

 

 

Data Mine News Headlines

Python offers some amazing tools to do text based searching, sorting and analysis.

In this blog I wanted to look at grabbing a large group of news headlines and then do some Natural Language Processing (NLP) in Python to try and find out “Who’s in the News”.

For this example I’ll be using the Reddit News, but any News feed like Twitter, CNN, BBC etc. could be used.

Reddit

Reddit is a social media source that is free to use. To pull information from Reddit you will need to create an account and get API client ID and secret. To do this go to: https://www.reddit.com/prefs/apps/ and select edit in the developed applications area.

reddit_api_info

The Reddit Python library is called Praw, and it is installed by:

pip install praw

An example to connect to the Reddit API and list the newest 4 News headlines would be:

# red1.py - Python Reddit Example 
# Get 4 Latest News Headlines
import praw

# Update with your client info
reddit = praw.Reddit(client_id='xQsMxxxxxxxx',
            client_secret='X8r62xxxxxxxxxxxx',
            user_agent='myreddit', username='yourname', password='xxxx')
					 
i=0					 
for submission in reddit.subreddit('news').new(limit=4):
	i += 1
	print(i,submission.title)

Running this code will show something like:

> python red1.py
1 De Blasio Unveils Health Care Plan for Undocumented and Low-Income New Yorkers
2 Kentucky teacher seen dragging student with autism down hall pleads not guilty
3 Homeless man allegedly involved in GoFundMe scam arrested in Philadelphia
4 Government shutdown stops FDA food safety inspections

The output from the program can be checked with the Reddit’s web page:

reddit_new4

When you are looking at Reddit it’s important to note that there are a number of different topics that could be queried. For example /inthenews is different than the /news.

Natural Language Processing

There are some amazing tools to allow you to manipulate and view the textual data. Pandas is a fast in memory data management library that supports sorting, querying and viewing of data. Spacy is a NLP tool. These two libraries are loaded by:

pip install pandas
pip install spacy

As a first example we’ll look at some text and we’ll use spacy to analyze what each word in the sentence is:

# Spacy test to get work types
#
import pandas as pd
import spacy

# Use the English core small web dictionary file
nlp = spacy.load('en_core_web_sm')
nlp.entity

# load some sample text into Spacy
doc = nlp('Astronomers in Canada have revealed details of 
       mysterious signals emanating from a distant galaxy')

print(doc,"\n")

# list the text and show the word type
for w in doc:
	print (w,w.pos_)

The output from this will:

>python red2.py
Astronomers in Canada have revealed details of mysterious signals emanating from a distant galaxy

Astronomers NOUN
in ADP
Canada PROPN
have VERB
revealed VERB
details NOUN
of ADP
mysterious ADJ
signals NOUN
emanating VERB
from ADP
a DET
distant ADJ
galaxy NOUN

Spacy will identify the words by their word type, like Astronomers NOUN. 

The proper nouns (PROPN) like Canada can be even further filtered to the type of noun, in this case location (GPE).

If you are only interested in proper nouns then it is possible to get the actual type of noun, for example: person, location, organization, work of art, date etc. To get the proper nouns the doc.ent object is queried.

# red3.py - Spacy test to get noun types
#
import pandas as pd
import spacy

nlp = spacy.load('en_core_web_sm')
nlp.entity

doc = nlp('Fred Smith and John Doe live in Toronto and they work for the Toronto Raptors.')

print(doc,"\n")

stuff = []
for w in doc.ents:
	print(w.text,w.label_)

The output for this is:

>python red3.py
Fred Smith and John Doe live in Toronto and they work for the Toronto Raptors.

Fred Smith PERSON
John Doe PERSON
Toronto GPE
the Toronto Raptors ORG

 

Pandas – to query/group and count

The Pandas library is extremely useful for doing statistical and data manipulation type functions.  If we expand our earlier example to include Pandas we can do some querying/grouping and counting

# NLP with Pandas data frames for queries/grouping/counting
#
import pandas as pd
import spacy

nlp = spacy.load('en_core_web_sm')
nlp.entity

doc = nlp('Fred Smith and John Doe live in Toronto and they work for the Toronto Raptors.')

print(doc,"\n")

stuff = []
for w in doc.ents:
	print(w.text,w.label_)
	stuff.append([w.text,w.label_])

# define a struction        
dflabel = ['keyword','wordtype']
# load a list into a Panda data frame with our structure
df = pd.DataFrame(stuff, columns=dflabel)

# print our data frame
print (df.head(n=50))

# create a new data frame with only the wordtype PERSON, then group and count it
names = df.query("wordtype=='PERSON'").groupby('keyword').count().sort_values(by='wordtype',ascending=False)

print (names.head(n=50))

The results for this would be:

Fred Smith and John Doe live in Toronto and they work for the Toronto Raptors.

Fred Smith PERSON
John Doe PERSON
Toronto GPE
the Toronto Raptors ORG

keyword wordtype
0 Fred Smith PERSON
1 John Doe PERSON
2 Toronto GPE
3 the Toronto Raptors ORG

          wordtype
keyword
Fred Smith 1
John Doe 1

Getting “Who’s in the News” from Reddit

Now we’re ready to put the pieces together. In this next example we’ll use the /inthenews Reddit section, and we’ll query 750 new items. From the results we’ll look at the people who are making the news.

# red_2_names.py - Get top names in Reddit "inthenews" 
#
import pandas as pd
import spacy
import praw

# Modify for your reddit id
reddit = praw.Reddit(client_id='xQsMfXXX',
                     client_secret='X8r62koQxxxxxxxx',
                     user_agent='myreddit', username='yourname', password='xxxx')
					 
thedata = ""
i=0
for submission in reddit.subreddit('inthenews').new(limit=750):
        i += 1
        #print(i,submission.title)
        thedata = thedata + submission.title
        
	
nlp = spacy.load('en_core_web_sm')
nlp.entity
doc = nlp(thedata)

# Create a list of keywords and wordtypes
stuff = []
dflabel = ['keyword','wordtype']
for w in doc.ents:
	stuff.append([w.text,w.label_])
#print(stuff)
	
df = pd.DataFrame(stuff, columns=dflabel)	
names = df.query("wordtype=='PERSON'").groupby('keyword').count().sort_values(by='wordtype',ascending=False)

print ("Who is making the news?\n")
print (names.head(n=10))

This example will generate results similar to the following:

python red_2_names.py
Who is making the news?

 wordtype
keyword
Trump 14
Ocasio-Cortez 6
Donald Trump 4
Cohen 3
Nancy Pelosi 2
Jeff Bezos 2
Ronald Reagan 2
Brown 2
Jayme Closs 2
Jared Kushner 2

The query could be changed to look at other nouns like locations, organization or all topics.

Final Comments

With just a small amount of code it is possible to do some amazing things. However like any statistical project we can improve the data set, for example entire news articles could be imported rather than just the headlines.

 

Android Python App Talking to a Raspberry Pi

If you’d like to use some of your Python experience on Android, then the QPython Android IDE may surprise you with what it has to offer.

Out of the box QPython includes some excellent libraries, especially if you’re interested in doing some data mining or modeling.

libraries

In this blog I’d like to look at an example of building an Android QPython GUI app that controls some lights on a Raspberry Pi.

A Button Interface

Luckily you can do all your development on a Windows/Linux/MacOS/Raspberry Pi before you need to move the application to Android.

QPython comes with a number of pre-installed libraries such as: a standalone Web Server, a low level Android interface, sockets, and the QPython supports the Pygame library.

I used a simple 4 button application, but there are lots of other graphic features that are possible in Pygame. The only real difference with a pygame QPython app is that a #qpy:pygame reference is required at the top of the file. This reference is overlooking in Window/Linux/MacOS.

#qpy:pygame

import pygame
import socket

pygame.init()

def draw_button(button, screen):
    #Draw the button rect and the text surface
    pygame.draw.rect(screen, button['color'], button['rect'])
    screen.blit(button['text'], button['text rect'])

def create_button(x, y, w, h, bg, text, callback):
    # Create a buttondictionary of the rect, text,
    # text rect, color and the callback function.
    FONT = pygame.font.Font(None, 50)
    text_surf = FONT.render(text, True, pygame.Color('black'))
    button_rect = pygame.Rect(x, y, w, h)
    text_rect = text_surf.get_rect(center=button_rect.center)
    button = {
        'rect': button_rect,
        'text': text_surf,
        'text rect': text_rect,
        'color': pygame.Color(bg),
        'callback': callback,
        }
    return button

def main():
    screen = pygame.display.set_mode((640, 480))
    pygame.display.set_caption("Rasp Pi Interface: ")
    clock = pygame.time.Clock()    

    def bt_func(input):  # A callback function for the button.
        pygame.display.set_caption("Rasp Pi Interface: Send - " + input)
        print(input)

    button1 = create_button(100, 50, 250, 80,'blue','BLUE LED',lambda: bt_func('blue'))
    button2 = create_button(100, 150, 250, 80,'yellow', 'YELOW LED',lambda:bt_func('yellow'))
    button3 = create_button(100, 250, 250, 80,'red', 'RED LED', lambda: bt_func('red') )
    button4 = create_button(100, 350, 250, 80,'green', 'GREEN LED',lambda: bt_func('green'))
    
    # A list that contains all buttons.
    button_list = [button1, button2,button3, button4]

    while True:
        for event in pygame.event.get():
            if event.type == pygame.QUIT:
                pygame.quit()
            # This block is executed once for each MOUSEBUTTONDOWN event.
            elif event.type == pygame.MOUSEBUTTONDOWN:
                # 1 is the left mouse button, 2 is middle, 3 is right.
                if event.button == 1:
                    for button in button_list:
                        # `event.pos` is the mouse position.
                        if button['rect'].collidepoint(event.pos):
                            # Increment the number by calling the callback
                            # function in the button list.
                            button['callback']()


        screen.fill(pygame.Color('white'))
        for button in button_list:
            draw_button(button, screen)
        pygame.display.update()
        clock.tick(30)

main()
pygame.quit()

When the Pygame application is running the buttons will send toggle the display caption and write a print() message.

rasp_app_linux

Socket Communications

Sockets are a simple way to have multiple devices communicate together. For this example the Raspberry Pi will be a Socket Server and a PC, Linux node or an Android phone could be socket clients.

# Simple Python Socket Server
#
import socketserver

class Handler_TCPServer(socketserver.BaseRequestHandler):
    # Handler to manage incoming requests
    def handle(self):
        # self.request - TCP socket connected to the client
        self.data = self.request.recv(1024).strip()
        # self.data - is the incoming message
        print("{} sent:".format(self.client_address[0]))
        print(self.data)
        # if required a request could be send back
        #self.request.sendall("ACK from TCP Server".encode())

if __name__ == "__main__":
    # Define the host and port to use
    HOST, PORT = "192.168.0.133", 9999

    # Init the TCP server object, bind it to the localhost on 9999 port
    tcp_server = socketserver.TCPServer((HOST, PORT), Handler_TCPServer)
    print("Socket Server Started on : " + HOST + " port: " + str(PORT))
    # Activate the TCP server.

    tcp_server.serve_forever()

A simple Python TCP socket client would be:

import socket

host_ip, server_port = "192.168.0.133", 9999
data = " Hello how are you?\n"

# Initialize a TCP client socket using SOCK_STREAM
tcp_client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

try:
    # Establish connection to TCP server and exchange data
    tcp_client.connect((host_ip, server_port))
    tcp_client.sendall(data.encode())

    # Read data from the TCP server and close the connection
    received = tcp_client.recv(1024)
finally:
    tcp_client.close()

print ("Bytes Sent:     {}".format(data))
print ("Bytes Received: {}".format(received.decode()))

Put It Together

Once the socket communications and the Pygame interface is working, the full program can be completed. The first step is to define some output LED pins on the Raspberry Pi. For my project I used a Pimoroni Explorer Hat Pro. This Pi top had some built in colored LED pins.

# Pi Socket Server to toggle 4 color LEDs
import socketserver

class Handler_TCPServer(socketserver.BaseRequestHandler):
    # Handler to manage incoming message requests

    def handle(self):
        # self.request - TCP socket connected to the client
        self.data = self.request.recv(1024).strip()
        print("{} sent:".format(self.client_address[0]))
        print(self.data)
        if self.data == "red":
            GPIO.output(redpin, not GPIO.input(redpin))
        if self.data == "yellow":
            GPIO.output(yellowpin,not GPIO.input(yellowpin))  
        if self.data == "blue":
            GPIO.output(bluepin,not GPIO.input(bluepin))
        if self.data == "green":
            GPIO.output(greenpin,not GPIO.input(greenpin))            
        # just send back ACK for data arrival confirmation
        #self.request.sendall("ACK from TCP Server".encode())

if __name__ == "__main__":
    # Define 4 LED pins as outputs
    import RPi.GPIO as GPIO
    GPIO.setwarnings(False)
    GPIO.setmode(GPIO.BCM)
    yellowpin = 17
    bluepin = 4
    redpin = 27
    greenpin = 5
    GPIO.setup(redpin,GPIO.OUT)
    GPIO.setup(yellowpin,GPIO.OUT)
    GPIO.setup(bluepin,GPIO.OUT)
    GPIO.setup(greenpin,GPIO.OUT)
     
    HOST, PORT = "192.168.0.133", 9999

    # Init the TCP server object, bind it to the localhost on 9999 port
    tcp_server = socketserver.TCPServer((HOST, PORT), Handler_TCPServer)
    print("Socket Server Started on : " + HOST + " port: " + str(PORT))
    # Activate the TCP server.
    # To abort the TCP server, press Ctrl-C.
    tcp_server.serve_forever()

Next our Pygame interface needs to include  some TCP socket client code that sends the color message.

#qpy:pygame

import pygame
import socket

pygame.init()

def draw_button(button, screen):
    #Draw the button rect and the text surface
    pygame.draw.rect(screen, button['color'], button['rect'])
    screen.blit(button['text'], button['text rect'])

def create_button(x, y, w, h, bg, text, callback):
    # Create a buttondictionary of the rect, text,
    # text rect, color and the callback function.
    FONT = pygame.font.Font(None, 50)
    text_surf = FONT.render(text, True, pygame.Color('black'))
    button_rect = pygame.Rect(x, y, w, h)
    text_rect = text_surf.get_rect(center=button_rect.center)
    button = {
        'rect': button_rect,
        'text': text_surf,
        'text rect': text_rect,
        'color': pygame.Color(bg),
        'callback': callback,
        }
    return button

def main():
    screen = pygame.display.set_mode((640, 480))
    pygame.display.set_caption("Rasp Pi Interface: ")
    clock = pygame.time.Clock()    

    def bt_func(input):  # A callback function for the button.
        pygame.display.set_caption("Rasp Pi Interface: Send - " + input)
        print(input)
        host_ip, server_port = "192.168.0.133", 9999
        tcp_client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        try:
        # Establish connection to TCP server and exchange data
            tcp_client.connect((host_ip, server_port))
            tcp_client.sendall(input.encode())
        finally:
            tcp_client.close()

    button1 = create_button(100, 50, 250, 80,'blue','BLUE LED',lambda: bt_func('blue'))
    button2 = create_button(100, 150, 250, 80,'yellow', 'YELOW LED',lambda: bt_func('yellow'))
    button3 = create_button(100, 250, 250, 80,'red', 'RED LED', lambda: bt_func('red'))
    button4 = create_button(100, 350, 250, 80,'green', 'GREEN LED',lambda: bt_func('green'))
    
    # A list that contains all buttons.
    button_list = [button1, button2,button3, button4]

    while True:
        for event in pygame.event.get():
            if event.type == pygame.QUIT:
                pygame.quit()
            # This block is executed once for each MOUSEBUTTONDOWN event.
            elif event.type == pygame.MOUSEBUTTONDOWN:
                # 1 is the left mouse button, 2 is middle, 3 is right.
                if event.button == 1:
                    for button in button_list:
                        # `event.pos` is the mouse position.
                        if button['rect'].collidepoint(event.pos):
                            # Increment the number by calling the callback
                            # function in the button list.
                            button['callback']()


        screen.fill(pygame.Color('white'))
        for button in button_list:
            draw_button(button, screen)
        pygame.display.update()
        clock.tick(30)

main()
pygame.quit()

Loading the Application on Android

To load the application on my phone, I connected my phone to a PC and I move the file to the phone’s /Download directory. QPython files can be created/edited and run using the Editor button. New Python applications can be created using the PygameApp menu option.

It’s important to note that if you are using print() functions in your application then a log directory file location prompt will come up the first time that your app is run.

QPython applications can be made into a desktop short cut.

Once everything is configured, you should be able to click on the desktop QPython icon and start controlling the Raspberry Pi.

android2pi

 

 

 

RabbitMQ for IoT

For Internet of Things (IoT) projects that are a lot of different ways that the sensors, devices and client interfaces can be connected together. For many projects using simple MQTT (Message Queue Telemetry Transport) is all that you need. However if you’re trying to merge and build IoT projects that use both MQTT and AMQP (Advanced Message Queue Protocol) or require a REST API then you should take a look at RabbitMQ.

RabbitMQ is an open source middleware solution that natively uses AMQP communications but it has a good selection of plug-ins to support features like: MQTT, MQTT Web Sockets, HTTP REST API and server-to-server communications.

rabbitmq_overview

In this blog we will setting up a RabbitMQ server, and we will look at some of the differences between MQTT and AMQP messaging. Finally an example of an Anduino MQTT message will be presented as both an MQTT and an AMQP item in a Node-Red dashboard.

Getting Started

RabbitMQ can be installed on Window, Linux, MacOS systems and there are also some cloud based offerings. For small systems lower end hardware like a Raspberry Pi can be used.  For complete RabbitMQ installation instructions see: https://www.rabbitmq.com/download.html . To install and run RabbitMQ on a Ubuntu system enter:

sudo apt-get update
sudo apt-get install rabbitmq-server
sudo service rabbitmq-server start

The next step is to add some plug-ins. For my project I loaded the MQTT and Web Administration plug-ins:

sudo rabbitmq-plugins enable rabbitmq_mqtt
sudo rabbitmq-plugins enable rabbitmq-management

The rabbitmqctl command line tool allows you to configure and review the RabbitMQ server. To add a user admin1,  with password admin1, that has config, write and read rights for management and administrator access, enter:

sudo rabbitmqctl add_user admin1 admin1
sudo rabbitmqctl set_permissions -p / admin1 ".*" ".*" ".*"
sudo rabbitmqctl set_user_tags admin1 management administrator

After you’ve defined an administrative user the RabbitMQ web management plug-in can be accessed by: http://ip_address:15672 .

Rabbit_webadmin

The RabbitMQ Web Management tool offers an overview of the present system load, connections, exchanges and queues. 

The RabbitMQ Web Management tool is excellent for small manual changes, if however you are looking a doing a large number of additions or changes then, rabbitmqadmin, the command line management tool can be used. This tool is  installed by:

# Get the cli and make it available to use.
wget http://127.0.0.1:15672/cli/rabbitmqadmin
sudo chmod +x rabbitmqadmin

Comparing MQTT and AMQP

It’s useful to comment about some of the differences between MQTT (Message Queue Telemetry Transport) and AMQP (Advanced Message Queueing Protocol) .

MQTT is a light weight publish-subscribe-based messaging protocol that works well with lower end hardware and limited bandwidth. For Arduino type applications where you only need to pass some sensor data MQTT is an excellent fit.

AMQP has more overhead than MQTT, because it is a more advanced protocol that includes message orientation, queuing, routing, reliability and security. Presently there are no mainstream AMQP Arduino libraries, but numerous programming options for Raspberry Pi, Linux, Windows and MacOS systems exist.  An AMQP IoT example would be to send sensor failures and alarms to dedicated maintenance and alarm queues.

MQTT and AMQP Queues

One of the biggest differences in queues is that MQTT queues are designed to show you the last available message, where as AMQP will store multiple messages in a queue.

A published MQTT message contains a message body, a retain flag and a quality of service (QoS) value.

An AMQP message can be published with added properties such as: time stamp, type of message and expiration information. AMQP messages also support the addition of custom header values. Below is a Python publish example that defines the message type to be “Pi Sensor”, and custom headers are include for status and alarm state.

#!/usr/bin/env python
import pika

node = "192.168.0.121"
user = "pete"
pwd = "pete"

# Connect to a remote AMQP server with a username/password
credentials = pika.PlainCredentials(user, pwd)
connection = pika.BlockingConnection(pika.ConnectionParameters(node,
        5672, '/', credentials))                                    
channel = connection.channel()

# Create a queue if it doesn't already exist
channel.queue_declare(queue='Rasp_1',durable=True)

# Define the properties and publish a message
props = pika.BasicProperties(
    headers= {'status': 'Good Quality',"alarm":"HI"},
    type ="Pi Sensor")
channel.basic_publish(exchange='',
    routing_key='Rasp_1',body='99.5', properties = props)

connection.close()

For this example the Rasp_1 queue can be examined using the Queue->Get Message option in the Web Management Interface. 

q_prop

RabbitMQ Messaging

There are a variety of ways that AMQP messages can be published and subscribed to. The simplest way is to create a queue and then messages can be published and subscribed to from that queue.

rabbitmq_layoutTo help with the distribution and filtering of messages AMQP supports a number of different exchange types. Messages in an exchange use bindings based on a routing key to link them to a queue. 

The main types of exchanges are: direct, fanout, headers and topic. An IoT example of a direct exchange would be if a group of Raspberry Pi sensor values are going into a “Rasp Pi Sensor” exchange. When the Rasp Pi publishes a sensor result to the exchange the message also includes a routing key to link the message to the correct queue.

direct

An IoT example of a fanout exchange would be with critical sensor failures. The sensor failure message is sent to Bill’s and Sam’s work or task queue and the All Maintenance point queue at the same time.

fanout

Connecting MQTT

After the MQTT plug-in is installed RabbitMQ can act like a standalone MQTT broker. MQTT data can also be made available through an AMQP subscription by binding the MQTT exchange to a RabbitMQ queue.

For an MQTT project any ESP8266 supported Arduino hardware can be used. There are a number of MQTT Arduino libraries that are available. For this project I used the PubSubClient that can be installed using the Arduino Library Manager.

As a test project I used at low cost MQ-2 Combustible Gas Sensor ($3) that measures a combination of LPG, Alcohol, Propane, Hydrogen, CO and even methane. Note to fully use this sensor some calibration is required. On the MQ-2 sensor the analog signal is connected to Arduino pin A0 and the analogRead(thePin) function is used to read the sensor value.

MPq2_Setup.png

Below is some example Arduino code required to read the MQ2 gas sensor and publish it to the RabbitMQ MQTT broker with a topic name of : mq2_mqtt.

/*
 Basic ESP8266 MQTT publish client example
*/
#include <ESP8266WiFi.h>
#include <PubSubClient.h>

// Update these with values suitable for your network.
const char* ssid = "your_ssid";
const char* password = "your_password";
const char* mqtt_server = "192.168.0.121"; 
const char* mqtt_user = "admin1";
const char* mqtt_pass= "admin1";

const int mq2pin = A0; //the MQ2 analog input pin

WiFiClient espClient;
PubSubClient client(espClient);

void setup_wifi() {
  // Connecting to a WiFi network
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}

void reconnect() {
  // Loop until we're reconnected
  Serial.println("In reconnect...");
  while (!client.connected()) {
    Serial.print("Attempting MQTT connection...");
    // Attempt to connect
    if (client.connect("Arduino_Gas", mqtt_user, mqtt_pass)) {
      Serial.println("connected");
    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
      delay(5000);
    }
  }
}

void setup() {
  Serial.begin(9600);
  setup_wifi();
  client.setServer(mqtt_server, 1883);
}

void loop() {
  char msg[8];
  if (!client.connected()) {
    reconnect();
  }

  sprintf(msg,"%i",analogRead(mq2pin));
  client.publish("mq2_mqtt", msg);
  Serial.print("MQ2 gas value:");
  Serial.println(msg);

  delay(5000);
}

Once the MQTT value is published any MQTT client can subscribe to it.  Below is a Python MQTT subscribe example.

# mqtt_sub.py - Python MQTT subscribe example 
#
import paho.mqtt.client as mqtt
 
def on_connect(client, userdata, flags, rc):
    print("Connected to broker")
 
def on_message(client, userdata, message):
    print ("Message received: "  + message.payload)

client = mqtt.Client()
client.username_pw_set("admin1", password='admin1')
client.connect("192.168.0.121", 1883) 

client.on_connect = on_connect       #attach function to callback
client.on_message = on_message       #attach function to callback

client.subscribe("mq2_mqtt") 
client.loop_forever()                 #start the loop

Read MQTT Messages Using AMQP

MQTT clients can subscribe to MQTT messages directly or it’s possible to configure RabbitMQ to have the MQTT data accessible as AMQP. The routing of MQTT messages to AMQP queues is done using the direct exchange method.

mqtt_2_amqp

To configure RabbitMQ to forward MQTT the following steps are done:

  1. Create a new RabbitMQ Queue – For an IoT project this would typically be a 1-to-1 mapping of the MQTT topic to a queue name.
  2. Create a binding between the MQTT Exchange and the Queue – by default all the MQTT topic go into the amq.topic exchange. For MQTT items in this exchange a binding, typically the MQTT topic name, is used as the routing key to the AMQP queue.

These steps can be done in a number of ways : manually, in the RabbitMQ config file, using the rabbitmqadmin command line tool or via a program. Because I was doing this for multiple signals I used rabbitmqadmin tool, and the syntax is:

./rabbitmqadmin declare queue name=mq2_amqp durable=true
./rabbitmqadmin declare binding source=amq.topic destination_type=queue destination=mq2_amqp routing_key=mq2_mqtt

The RabbitMQ Web Admin can be used to verify the exchange to queue binding.

wp_mqtt_2_amqp

 

The CLI tool can also be used to see if there are any values in the queue:

cli_get_queue

Node-Red Dashboard

Node-Red is a visual programming environment that allows users to create applications by dragging and dropping nodes on the screen. Logic flows are then created by connecting the different nodes together.

Node-Red has been preinstalled on Raspbian Jesse since the November 2015. Node-Red can also be installed on Windows, Linux and OSX.  To install and run Node-Red on your specific system see https://nodered.org/docs/getting-started/installation.

To install the AMQP components, select the Manage palette option from the right side of the menu bar. Then search for “AMQP” and install node-red-contrib-amqp. If your installation of Node-Red does not have dashboards installed, search for: node-red-dashboard.

nr_install_amqp

For our Node-Red MQTT and AMQP example we will use a mqtt and a amqp node from the input palette group, and two gauge nodes from the dashboard group.  The complete Node-Red logic for this is done in only 4 nodes!

nr_logic

Nodes are added by dragging and dropped them into the center Flow sheet. Logic is created by making connection wires between inputs and output of a node. After the logic is laid out, double click on each of the nodes to configure their specific properties. You will need to specify the MQTT and AMQP definitions of your RabbitMQ IP address,user rights, MQTT topic and AMQP queue name. You will also need to double click on the gauge nodes to configure the look-and-feel of the web dashboard.

After the logic is complete, hit the Deploy button on the right side of the menu bar to run the logic.  The Node-Red dashboard user interface is accessed by: http://ipaddress:1880/ui. 

For my project I used a number of different MQ sensors and inputs. Below is a picture of Node-Red web dashboard that we created with the same MQTT value being shown natively and as a AMQP queued value.

project

Final Comments

I found that RabbitMQ was easy to install and the Web Administration plug-in along with the rabbitmqadmin command line tool made it very easy to maintain. 

If you’re just looking to show sensor values then a basic MQTT broker might be all you need. However if you’re looking at some future applications like alarm, maintenance or task lists then AMQP exchanges and queue make RabbitMQ an interesting option.

IoT with Google Firebase

For many Internet of Things (IoT) projects a message queuing system like MQTT (Message Queue Telemetry Transport) is all that is needed to connect sensors, devices and graphic interfaces together. If however you require a database, with sorting, queuing and multi-media support then there are some great cloud storage platforms that are available. One that is definitely worth taking a look at is Google Firebase.

Like any IoT solution, Google Firebase can have inputs and sensors sending data directly into it, and a variety of client applications to view the data (Figure 1), but Google Firebase also offers other features such as: file storage, machine learning, messaging, and server side functions. In this article I will look at:

In this article I will look at:

  1. Setting up a sample Firebase IoT project
  2. Use Python to simulate I/O data
  3. Create a Web dashboard with Node-Red to view and write data
  4. Create an Android app using AppInventor to view and write data
  5. Look at a more complex data monitoring example in Python

Getting Started

Google Firebase [1] may not have the huge variety of options that Amazon Web Services (AWS) has, but I found as an IoT engineer Google Firebase had all the features that I needed to get up and running quickly, and I didn’t need a credit card for my free activation.

To begin log in with your Google account at https://firebase.google.com/, and select “Get Started”. I created a sample project called My-IoT. After the project is created Firebase will give it a unique identifier name, such as: my-iot-4ded7.

Firebase data can be stored in either a Realtime Database, or a Cloud Firestore. The Realtime Database is an unstructured NoSQL format that has been around for about 4 years, so there are lots of 3rd party components. The Cloud Firestore stores the data in structure collections and it’s fairly new so the 3rd party options are a little more limited.

For our test IoT project we will use the  real-time database, and it can be created from the Database -> Data menu option.

rtdb

The database structure can be built directly from the Firebase web console or imported from a JSON file. For this database 3 grouping are defined, one for Android, Node-Red and Raspberry Pi I/O values. The Firebase console allows for setting, changing and viewing the database values.

db_struct

The security is configured in the Database -> Rules menu option. To keep things simple for testing read and write security is set to true for public access,  (remember to change this for a production system).

security

The project’s remote access settings are shown in the Authentication section with the “Web setup” button.

webconfig

Python and Firebase

There are a number of Python libraries to access Firebase. The pyrebase library has some added functionality such as: queries, sorting, file downloads and streaming support. The pyrebase library only support Python 3, and it is installed by:

sudo pip3 install pyrebase
sudo pip3 install --upgrade google-auth-oauthlib

A Python 3 test program to set two values (pi/pi_value1 and pi/pi_value2) and read a single point and a group of point is shown below. Remember to change the configuration settings to match your project settings.

import pyrebase, random

# define the Firebase as per your settings
config = {
  "apiKey": "AIzaSyCq_WytLdmOy1AIzaSyCq_WytLdmOy1",
  "authDomain": "my-iot-4ded7.firebaseapp.com",
  "databaseURL": "https://my-iot-4ded7.firebaseio.com",
  "storageBucket": "my-iot-4ded7.appspot.com"
}

firebase = pyrebase.initialize_app(config)
db = firebase.database()

# set 2 values with random numbers
db.child("pi").child("pi_value1").set(random.randint(0,100))
db.child("pi").child("pi_value2").set(random.randint(0,100))

# readback a single value
thevalue = db.child("pi").child("pi_value1").get().val()
print ("Pi Value 1: ", thevalue)

# get all android values
all_users = db.child("android").get()
for user in all_users.each():
    print(user.key(), user.val())

The Firebase web console can be used to check the results from the test program.

Node-Red and Firebase

Node-Red is a visual programming environment that allows users to create applications by dragging and dropping nodes on the screen. Logic flows are then created by connecting the different nodes together.

Node-Red has been preinstalled on Raspbian Jesse since the November 2015. Node-Red can also be installed on Windows, Linux and OSX.  To install and run Node-Red on your specific system see https://nodered.org/docs/getting-started/installation.

To install the Firebase components, select the Manage palette option from the right side of the menu bar. Then search for “firebase” and install node-red-contrib-firebase.

nr_fb_install

For our Node-Red example we will use web gauges to show the values of our two Python simulated test points. We will also send a text comment back to our Firebase database. The complete Node-Red logic for this is done in only 6 nodes!

nr_logic

To read the Pi values two firebase.on() nodes are used. The output from these nodes is connected to two dashboard gauge nodes. Double-clicking on firebase_on() node to configure Firebase database and the item to read from. Double-clicking on the gauge node allows you to edit gauge properties.

nr_value1 To send a text string to Firebase a text input node is wired to a firebase modify node. Edit the firebase modify node with the correct database address and value to set.

nr_modify

After the logic is complete, hit the Deploy button on the right side of the menu bar to run the logic.  The Node-Red dashboard user interface is accessed by: http://ipaddress:1880/ui, so for example 192.168.1.108:1880/ui.

nr_screen

AppInventor

AppInventor is a Web based Android app creation tool (http://appinventor.mit.edu), that uses a graphical programming environment.

AppInventor has two main screens. The Designer screen is used for the layout or presentation of the Android app and the Blocks screen is used to build the logic. On the right side of the top menu bar, the Designer and Blocks buttons allow you to toggle between these two screens.

On the Designer screen, an app layout is created by dragging a component from the Palette window onto the Viewer window.

We will use AppInventor to create an Android app that will read Pi values from our Firebase IoT database, and write a value back.

For the visuals on this application a Button, Label (2),  ListView, Textbox and FirebaseDB component will be used. After a component is added to the application, the Components window is used to rename or delete that component. The Properties window is used to edit the features on a component. For the FirebaseDB component configure the Token and URL to match with your project settings.

fb_app_setup

Once the layout design is complete, logic can be added by clicking on the Blocks button on the top menu bar.

Logic is built by selecting an object in the Blocks window, and then click on the specific block that you would like to use.

AppInventor is pretty amazing when it comes to doing some very quick prototyping. Our entire app only uses two main blocks.

The when FirebaseDB1.DataChanged block is executed whenever new data arrives into the Firebase database. The DataChanged block returns a tag and a value variable. The tag variable is the top level item name, (“pi” or “nodered” for our example). The value will be a string of the items and their values, for example: “{pi_value2 = 34, pi_value1=77}”. We use a replace all text block to remove the “{” and “}” characters, then we pass the string to the ListView component. Note this same code will pass 2 or 200 tags in the pi tag section.

The when BT_PUT.click block will pass the text that we enter on the screen into the android/and_value1 item in our database.

fb_app_code

After the screen layout and logic is complete, the menu item Build will compile the app. The app can be made available as an APK downloadable file or as a QR code link.

Below is picture of the final Android app synced with the Firebase data.

Firebase2

Python Data Monitoring Example

Our starting IoT Firebase realtime database was quite simple. Data monitoring or SCADA (Supervisory Control and Data Aquisition) projects usually require more information than a just a value. A more realistic sensor database would includes fields such as tag name, description, status, time and units.

fb_db2

By adding some indexing in the Firebase Security Rules it is possible to create some queries and sorts of the data. Some typical queries would be: “Points in alarm” or “Point values between 2:00 and 2:15”.

fb_rules2

In the code listing below the syntax statement of .order_by_child(“status”).equal_to(“ALARM”).get() is used to show only records that have a status = ALARM. Some other filter options include: .start_at(time1).end_at(time2).get, .limit_to_first(n), and .order_by_value().

A good next step would be to pass important filtered information to Google Firebase’s messaging feature.

import pyrebase, random

# define the Firebase as per your settings
config = {
  "apiKey": "AIzaSyCq_AIzaSyCq_Wyt",
  "authDomain": "my-iot-7ded7.firebaseapp.com",
  "databaseURL": "https://my-iot-7ded7.firebaseio.com",
  "storageBucket": "my-iot-7ded7.appspot.com"
}

firebase = pyrebase.initialize_app(config)
db = firebase.database()

tag_sum = db.child("pi").order_by_child("status").equal_to("ALARM").get()

# print alarm points
for tag in tag_sum.each():
    info = tag.val()
    print (tag.key()," - ", info["description"],":", info["value"],info["units"], info["status"])

For more Python examples see the Pyrebase documentation.

Summary

Without a lot of configuration it is possible to configure a Google Firebase project and have Python, Node-Red and Android app read and write values. The level of programming complexity is on par with an MQTT implementation, however Google Firebase can offer a lot more future functionality that you wouldn’t have with MQTT, such as file storage, machine learning, messaging, and server side functions.

At the time of writing this article I found the Arduino Firebase library to be unreliable but Google offers some options like an MQTT bridge.