Pi Appliance

My goal was to make a Pi kitchen appliance that shows me the key things that I want to see and music I want to listen to while I’m getting my morning coffee. For this project I used a Rasp Pi with 2.8″ TFT touchscreen. These screens start at a round $15.

People’s morning interests will vary so in this blog I just wanted to highlight some of the issues that I needed to worked through. For me the main stumbling blocks were:

  • Hiding the top Rasp Pi menu bar
  • Creating a GUI that uses the full screen
  • Getting weather data
  • scraping web pages to extract what I need

Getting Started

There are some great Raspberry Pi TFT screens that come with buttons and cases. You will need to look at the documentation that comes with your screen, but a good reference is: https://learn.adafruit.com/adafruit-pitft-28-inch-resistive-touchscreen-display-raspberry-pitft_case

For my project I simply used some of my kids Lego blocks.

pi_kitch2

Remove the Pi Menu Bar

The Pi TFT screen isn’t super large, so I wanted to remove the Pi menu bar and run my application at full size.

tft_w_menu

To remove the menu bar tweek two files. First:

sudo nano /etc/xdg/lxsession/LXDE-pi/autostart

Comment out the line  (with #) :

@lxpanel --profile LXDE

Then do the same for:

nano /home/pi/.config/lxsession/LXDE-pi/autostart

After this reboot the system.

Create a Full Size App

There are a multitude of choices for a screen layout. I was looking for lines of text, with maybe the bottom line used for buttons. I found that 7 lines was a reasonable fit. To remove the Python Tkinter title I positioned the top of the screen above the physical screen position (-30 instead of 0).


# My Kitchen Appliance App
#
import urllib.request as urllib2
import tkinter as Tkinter
from tkinter.ttk import *

from tkinter.font import Font
from tkinter import messagebox
top = Tkinter.Tk()
top.title("My Kitchen Appliance")
top.geometry("320x240+-5+-30") # set screen size, left (-5) and top (-30)
top.resizable(False, False)
top.details_expanded = False

#Define the buttons
myfont = Font(family="Times New Roman Bold",size= 12) # Should try a few more sizes

tft_rows = 7 # try 7 rows of buttons
tftbutton = ['' for i in range(tft_rows)]
for i in range(tft_rows):
    tftbutton[i] = Tkinter.Button(top, text = "Line " + str(i+1), fg = "blue", bg = "white", anchor="w", width= 35, height= 1,font=myfont).grid(row=(i+1),column=1) # a buttpn arra

top.mainloop()

The Python GUI will look like this:

tft_7bttns

Get Weather Data

There are a number of good weather API’s. I used OpenWeather because I can use it in variety of apps like Node-Red. OpenWeather has a free user API but you should login and get an appid.

A Python example to get some current weather data for a city:


# get Open Weather (REST API) data
import requests

# api-endpoint

URL = "https://openweathermap.org/data/2.5/weather?q="
mycity = "burlington,CA"
myappid = "&appid=b6907d289e10d714a6e88b30761fae22"
# sending get request and saving the response as response object
fullURL = URL + mycity + myappid
r = requests.get(fullURL)

# extracting data in json format
data = r.json()

print (data)

# Check out the structure
#for index, value in enumerate(data):
# print(index, value)

# Show some weather data
print (data['weather'][0]['description'])
print (data['weather'][0]['main'])
print (str(int(data['main']['temp'])) + " C")
# convert wind speed from meters/sec to kph
print (str((data['wind']['speed'] * 3.6)) + " kph")

This code will give output such as:

$Python3 burlweather.py
{'coord': {'lon': -79.8, 'lat': 43.32}, 'weather': [{'id': 803, 'main': 
'Clouds', 'description': 'broken clouds', 'icon': '04n'}], 'base': 
'stations', 'main': {'temp': 5.81, 'pressure': 1014, 'humidity': 93, 
'temp_min': 3.33, 'temp_max': 7.78}, 'visibility': 24140, 'wind': 
{'speed': 2.1, 'deg': 50}, 'clouds': {'all': 75}, 'dt': 1574816701,
 'sys': {'type': 1, 'id': 818, 'country': 'CA', 'sunrise': 1574771158, 
'sunset': 1574804839}, 'timezone': -18000, 'id': 5911592, 'name': 'Burlington', 'cod': 200}
broken clouds
Clouds
5 C
7 kph

Scraping Web Pages

I wasn’t able to find an API for all the things I was after, so I need to scrape web pages. The Python Beautiful Soup library is a great for finding and grabbing stuff on web pages. To install it:

$ apt-get install python-bs4 (for Python 2)

$ apt-get install python3-bs4 (for Python 3)

I had an example where I wanted to find the ski lifts and runs open. I had the Web page but I needed to search the ugly HTML code.

ski_bs0

ski_bs

In the HTML code I found that the lift and run information is contained in a <p class=“open value” tag. Beautiful Soup allows you to make searches based on attributes. The results can be HTML code or the .text property will return the results as simple text (no HTML code).

The following Python code would search my URL and extract the number of lifts open:


$ python
Python 3.7.4
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.request as urllib2
>>> from bs4 import BeautifulSoup
>>> theurl = 'https://www.onthesnow.ca/ontario/blue-mountain/skireport.html'
>>> page = urllib2.urlopen(theurl)
>>> soup = BeautifulSoup(page, 'html.parser')
>>> liftopen = soup.find("p", attrs={"class":"open value"})
>>> liftopen.text
'2 of 11'

Final Comments

There are a ton of different “Pi Appliance” applications that could be done. I hope that some of these hints that I’ve documented are helpful.

pi_kitch1

Data Mine News Headlines

Python offers some amazing tools to do text based searching, sorting and analysis.

In this blog I wanted to look at grabbing a large group of news headlines and then do some Natural Language Processing (NLP) in Python to try and find out “Who’s in the News”.

For this example I’ll be using the Reddit News, but any News feed like Twitter, CNN, BBC etc. could be used.

Reddit

Reddit is a social media source that is free to use. To pull information from Reddit you will need to create an account and get API client ID and secret. To do this go to: https://www.reddit.com/prefs/apps/ and select edit in the developed applications area.

reddit_api_info

The Reddit Python library is called Praw, and it is installed by:

pip install praw

An example to connect to the Reddit API and list the newest 4 News headlines would be:

# red1.py - Python Reddit Example 
# Get 4 Latest News Headlines
import praw

# Update with your client info
reddit = praw.Reddit(client_id='xQsMxxxxxxxx',
            client_secret='X8r62xxxxxxxxxxxx',
            user_agent='myreddit', username='yourname', password='xxxx')
					 
i=0					 
for submission in reddit.subreddit('news').new(limit=4):
	i += 1
	print(i,submission.title)

Running this code will show something like:

> python red1.py
1 De Blasio Unveils Health Care Plan for Undocumented and Low-Income New Yorkers
2 Kentucky teacher seen dragging student with autism down hall pleads not guilty
3 Homeless man allegedly involved in GoFundMe scam arrested in Philadelphia
4 Government shutdown stops FDA food safety inspections

The output from the program can be checked with the Reddit’s web page:

reddit_new4

When you are looking at Reddit it’s important to note that there are a number of different topics that could be queried. For example /inthenews is different than the /news.

Natural Language Processing

There are some amazing tools to allow you to manipulate and view the textual data. Pandas is a fast in memory data management library that supports sorting, querying and viewing of data. Spacy is a NLP tool. These two libraries are loaded by:

pip install pandas
pip install spacy

As a first example we’ll look at some text and we’ll use spacy to analyze what each word in the sentence is:

# Spacy test to get work types
#
import pandas as pd
import spacy

# Use the English core small web dictionary file
nlp = spacy.load('en_core_web_sm')
nlp.entity

# load some sample text into Spacy
doc = nlp('Astronomers in Canada have revealed details of 
       mysterious signals emanating from a distant galaxy')

print(doc,"\n")

# list the text and show the word type
for w in doc:
	print (w,w.pos_)

The output from this will:

>python red2.py
Astronomers in Canada have revealed details of mysterious signals emanating from a distant galaxy

Astronomers NOUN
in ADP
Canada PROPN
have VERB
revealed VERB
details NOUN
of ADP
mysterious ADJ
signals NOUN
emanating VERB
from ADP
a DET
distant ADJ
galaxy NOUN

Spacy will identify the words by their word type, like Astronomers NOUN. 

The proper nouns (PROPN) like Canada can be even further filtered to the type of noun, in this case location (GPE).

If you are only interested in proper nouns then it is possible to get the actual type of noun, for example: person, location, organization, work of art, date etc. To get the proper nouns the doc.ent object is queried.

# red3.py - Spacy test to get noun types
#
import pandas as pd
import spacy

nlp = spacy.load('en_core_web_sm')
nlp.entity

doc = nlp('Fred Smith and John Doe live in Toronto and they work for the Toronto Raptors.')

print(doc,"\n")

stuff = []
for w in doc.ents:
	print(w.text,w.label_)

The output for this is:

>python red3.py
Fred Smith and John Doe live in Toronto and they work for the Toronto Raptors.

Fred Smith PERSON
John Doe PERSON
Toronto GPE
the Toronto Raptors ORG

 

Pandas – to query/group and count

The Pandas library is extremely useful for doing statistical and data manipulation type functions.  If we expand our earlier example to include Pandas we can do some querying/grouping and counting

# NLP with Pandas data frames for queries/grouping/counting
#
import pandas as pd
import spacy

nlp = spacy.load('en_core_web_sm')
nlp.entity

doc = nlp('Fred Smith and John Doe live in Toronto and they work for the Toronto Raptors.')

print(doc,"\n")

stuff = []
for w in doc.ents:
	print(w.text,w.label_)
	stuff.append([w.text,w.label_])

# define a struction        
dflabel = ['keyword','wordtype']
# load a list into a Panda data frame with our structure
df = pd.DataFrame(stuff, columns=dflabel)

# print our data frame
print (df.head(n=50))

# create a new data frame with only the wordtype PERSON, then group and count it
names = df.query("wordtype=='PERSON'").groupby('keyword').count().sort_values(by='wordtype',ascending=False)

print (names.head(n=50))

The results for this would be:

Fred Smith and John Doe live in Toronto and they work for the Toronto Raptors.

Fred Smith PERSON
John Doe PERSON
Toronto GPE
the Toronto Raptors ORG

keyword wordtype
0 Fred Smith PERSON
1 John Doe PERSON
2 Toronto GPE
3 the Toronto Raptors ORG

          wordtype
keyword
Fred Smith 1
John Doe 1

Getting “Who’s in the News” from Reddit

Now we’re ready to put the pieces together. In this next example we’ll use the /inthenews Reddit section, and we’ll query 750 new items. From the results we’ll look at the people who are making the news.

# red_2_names.py - Get top names in Reddit "inthenews" 
#
import pandas as pd
import spacy
import praw

# Modify for your reddit id
reddit = praw.Reddit(client_id='xQsMfXXX',
                     client_secret='X8r62koQxxxxxxxx',
                     user_agent='myreddit', username='yourname', password='xxxx')
					 
thedata = ""
i=0
for submission in reddit.subreddit('inthenews').new(limit=750):
        i += 1
        #print(i,submission.title)
        thedata = thedata + submission.title
        
	
nlp = spacy.load('en_core_web_sm')
nlp.entity
doc = nlp(thedata)

# Create a list of keywords and wordtypes
stuff = []
dflabel = ['keyword','wordtype']
for w in doc.ents:
	stuff.append([w.text,w.label_])
#print(stuff)
	
df = pd.DataFrame(stuff, columns=dflabel)	
names = df.query("wordtype=='PERSON'").groupby('keyword').count().sort_values(by='wordtype',ascending=False)

print ("Who is making the news?\n")
print (names.head(n=10))

This example will generate results similar to the following:

python red_2_names.py
Who is making the news?

 wordtype
keyword
Trump 14
Ocasio-Cortez 6
Donald Trump 4
Cohen 3
Nancy Pelosi 2
Jeff Bezos 2
Ronald Reagan 2
Brown 2
Jayme Closs 2
Jared Kushner 2

The query could be changed to look at other nouns like locations, organization or all topics.

Final Comments

With just a small amount of code it is possible to do some amazing things. However like any statistical project we can improve the data set, for example entire news articles could be imported rather than just the headlines.