Memcached: fast lightweight network cache

For many projects an SQL database is overkill for simple storage of values. As an alternate solution there are a number of excellent distributed in-memory caching systems that can be used.

Two of the most popular in-memory caching systems are Redis and Memcached.

I’m big fan of Redis, and I’ve enjoyed doing projects with it. Redis offer some awesome speed with a small footprint, and it has many features that make it even superior to a messaging system like MQTT.

However if you’re looking for something 100% dead simple you should take a look a memcached, it has a super simple setup and with only a dozen commands so you’ll get up and running in no time.

There are API’s in all the common programming languages. It only takes 1 line of Bash to read or write to memchached.

Getting Started

Memached can be installed on all major OS’s. To install it on Ubuntu/Raspberry Pi:

sudo apt-get install memcached

If you’re using Docker there are some lightweight memcached images (89MB) that can be used. To run the memcached docker image the –net host option should be used:

$ sudo docker run -it --net host memcached

The -it (interactive) option could be useful if there are any errors kicked out.

See the man pages for a full description of memcached options.

Telnet can be used to enter manual enter commands (port 11211 is the default):

$ telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
version
VERSION 1.6.12

lru_crawler metadump all
key=mynum exp=-1 la=1642363105 cas=2 fetch=no cls=1 size=66
END
quit
Connection closed by foreign host.

For more information on the commands and Telnet.

Bash Interface to Memcached

There are lot of help guides for other Python, PHP, GO etc., but not a lot on how to use Bash.

The Bash nc command can be used to read and write to socket. The -q option will close the socket after 0 seconds (after the echo command is sent). The default port is 11211 but this can be changed along with adding some security options. The first example is to use the stats command:

$ echo "stats" | nc -q 0  192.168.0.120 11211 
STAT pid 1
STAT uptime 722
STAT time 1642359428
STAT version 1.6.12
...
END

$ # Find the number of current key items
$ echo "stats" | nc -q 0  192.168.0.120 11211 | grep curr_items
STAT curr_items 1

Set/Get a Key and Value

To set an in-memory key-value store, the syntax is:

set mykey <flags> <ttl> <size>
value

The flags option is typical set to 0. The ttl “time to live” is in seconds, a ttl of 0 is indefinite.The size of the value also needs to be define. (This is taken care of in the Python, PHP… libraries). It’s important to note that a newline (\r\n) is required before the value.

To set a key to a variable mynum with a value of 55 and a indefinite time to live:

$ # hard code the arguments in a set key command
$ # Note: echo -e is used to pass the \r\n for new lines
$ echo -e "set mynum 0 0 2 \r\n55\r" | nc -q 0  127.0.0.1 11211
STORED

A more flexible approach would be to pass variables:

$ # Set a new key/value pair, with no flags and indefinite timeout
$ ipp="192.168.0.120 11211"
$ thekey="mykey1"
$ thenum=55
$ numsize=${#thenum}
$ # send command 
$ echo -e "set $thekey 0 0 $numsize\r\n$thenum\r" | nc -q 0  $ipp
STORED

Get a key/value

The get key command returns 3 lines with the value being the 2nd line. Some awk code can be used to parse out just the value.

$ # Get a new key/value pair
$ ipp="192.168.0.120 11211"
$ thekey="mykey1"
$ 
$ echo "get $thekey" | nc -q 0  $ipp
VALUE mykey1 0 2
60
END
$ # Get just the 2nd line with the value
$ echo "get $thekey" | nc -q 0  $ipp | awk '{if (NR == 2) print $0}'
55
$ 
$ # Store the result in a variable
$ mykey1=$(echo "get $thekey" | nc -q 0  $ipp | awk '{if (NR == 2) print $0}')
$ echo "mykey1 = $mykey1"
mykey1 = 55

Increment/Decrement a Value

The inc / decr commands will increase or decrease a numeric stored key value by a defined amount:

$ #incr/decr a value's number
#
ipp="192.168.0.120 11211"
thekey="mykey1"

# Get starting value
echo "get mynum" | nc -q 0 $ipp | awk '{if (NR == 2) print $0}'
55

thediff=10; # increase the value (only positive values)
# incr and show the value
echo "incr $thekey $thediff" | nc -q 0  $ipp
65

thediff=5; # decrease the value (only positive values)
# incr and show the value
echo "decr $thekey $thediff" | nc -q 0 $ipp
60

Prepending and Appending

Memcached does not have queue or list functionality, if you need this take a look at Redis.

Below is an example script that creates a diagnostic log in a key/value. The append command adds msg text to the end of the overall string.

#!/usr/bin/bash
#
# diagmsg.sh - append msgs to a memcached variable string
#
ipp="192.168.0.120 11211"
msg="1:00 - Base Software Loaded\r"
size=${#msg}

echo -e "set mymsg 0 0 $size\r\n$msg\n\r" | nc -q 0  $ipp

# Create an array of diagnostics
diagmsgs=("2:00 - System Started\r" "2:15 - Getting Data\r" "2:30 - Backing up\r") 
# Append array to value text
for msg in "${diagmsgs[@]}"
do
  size=${#msg}
  #echo "Size: $size $msg";
  echo -e "append mymsg 0 0 $size\r\n$msg\n\r" | nc -q 0  $ipp;
done

# Show the result
echo -e "\nDiagnostic Results\n" 
echo "get mymsg" | nc -q 0 $ipp 

The results of this script would be:

$ bash diagmsg.sh
STORED
STORED
STORED
STORED

Diagnostic Results

VALUE mymsg 0 92
1:00 - Base Software Loaded
2:00 - System Started
2:15 - Getting Data
2:30 - Backing up

END

Final Thought

I played with the Python and PHP library and they were quite easy to use.

By adding a Bash interface to memcache it allows me to use programs like Octave/Matlib where a native interface isn’t available (just use the System call and pass the Bash code).

Sample Code: PHP

PHP example:

<?php
// memc.php - test memcached
//
// Load the PHP memcached component:
//
//    sudo apt-get install php-memcached

$memcache = new Memcached();
$memcache->addServer('localhost', 11211) or die ("Could not connect");

$version = $memcache->getVersion();
var_dump( $version);

$tmp_object = new stdClass;
$tmp_object->str_attr = 'test';
$tmp_object->int_attr = 123;

$memcache->set('key', $tmp_object);
echo "Store data in the cache (data will expire in 10 seconds)<br/>\n";

$get_result = $memcache->get('key');
echo "Data from the cache:<br/>\n";

var_dump($get_result);

?>

Sample Code: Python

Python example:

$ # Install Python memcache library
$ python3 -m pip install pymemcache

$ # Example set/get
$ python3
Python 3.8.10 (default, Nov 26 2021, 20:14:08) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pymemcache.client import base
>>> client = base.Client(('localhost', 11211))
>>> client.set('mykey','teststring')
True
>>> client.get('mykey')
b'teststring'
>>> client.set('mynum',44)
True
>>> client.get('mynum')
b'44'

Octave: A Free Matlab Alternative

Matlab has been an extremely popular tool for Engineering and Mathematical problem solving. Matlab has a wide range of practical applications and use-cases. It is also used in complex industrial applications such as multi-variable control in chemical reactors.

Matlab is a full featured commercial product that has a perpetual license for $2,150 US (or $860 yearly). Student pricing, (starting at $49 US, add-ons extras ), home pricing ($149 US, add-ons extra) and 30 day trials are available for non-commercial projects.

Octave is an open source alternative to Matlab that runs on GNU/Linux, macOS, BSD, and Microsoft Windows. The Octave syntax is largely compatible with Matlab.

I wish I had known about Octave when I was trying to help my kids with their high school Algebra and Calculus homework. In this blog I was look at introducing Octave with some high school math questions. I will finish with a quick data analysis example and a sensor example.

Getting Started with Octave

Octave/Matlab has a lot of potential applications and it can be loaded on a standard PC for math/stats projects or on a Raspberry Pi for projects that use GPIO (General Purpose Inputs and Outputs) to connect to sensors and I/O devices.

See the Octave web site (https://www.gnu.org/software/octave/download) for installation details for your specifics system. To load Octave on a Raspberry Pi/Ubuntu/Debian:

# load the base Octave software
sudo apt-get install octave
# add some of the key Octave packages
sudo apt-get install octave-control octave-image octave-io octave-optim octave-signal octave-statistics

The Octave Forge site (https://octave.sourceforge.io/packages.php ) has a list of useful community based libraries that can be added. To load a custom library, first install any dependencies then use the Octave pkg (package) command to install and load new library. For example to load the Symbolic library:

$ # install package dependencies
$ pip3 install sympy==1.5.1
$ octave
octave:1> # install symbolic package from forge site
octave:1> pkg install -forge symbolic
octave:2> # make package available by loading
octave:2> pkg load symbolic
octave:3> # show installed and loaded (*) packages
octave:3> pkg list
Package Name  | Version | Installation directory
--------------+---------+-----------------------
     control  |   3.2.0 | /usr/share/octave/packages/control-3.2.0
   dataframe  |   1.2.0 | /home/pete/octave/dataframe-1.2.0
       image  |  2.12.0 | /usr/share/octave/packages/image-2.12.0
          io  |  2.4.13 | /usr/share/octave/packages/io-2.4.13
       optim  |   1.6.0 | /usr/share/octave/packages/optim-1.6.0
      signal  |   1.4.1 | /usr/share/octave/packages/signal-1.4.1
  statistics  |   1.4.1 | /usr/share/octave/packages/statistics-1.4.1
      struct  |  1.0.16 | /usr/share/octave/packages/struct-1.0.16
    symbolic *|   2.9.0 | /home/pete/octave/symbolic-2.9.0

Octave GUI

Octave can be run in either a command line or a GUI mode.

The command line mode allows for basic testing and it is good for slower hardware like a Raspberry Pi 3.

The Octave GUI, which is called in Linux by:

$ octave --gui 

The Octave GUI offers a development environment with a script debugger and online help.

Solve Quadratic Equations

The power of Octave or Matlab can shown in it’s handling of Algebraic equations.

A typical high school math question would be to find where 2 equations intersect. Below is an example to solve where:

2*x^2 - 8*x - 4 = 3*x -x^2

Some comments on Octave/Matlab syntax, an exponent is define by: “.^” , and a “==” denotes that symbolic equation1 = equation2. A single “=” sets a variable to a value. If a line ends in “;” the lines results are hidden, otherwise the lines results are shown. The Octave prompt can be simplified by the command: PS1(“>> #”) .

>> #Solve for the intercept between 2 equations
>> pkg load symbolic
>> syms x
>> eq1 = 2*x.^2 - 8*x - 4 == 3*x -x.^2
eq1 = (sym)

     2                2      
  2⋅x  - 8⋅x - 4 = - x  + 3⋅x

>> solve(eq1)
ans = (sym 2×1 matrix)

  ⎡-1/3⎤
  ⎢    ⎥
  ⎣ 4  ⎦
>> # Show the number numerically
>> double(ans)
ans =

  -0.33333
   4.0
>># Plot the first equation
>> ezplot(@(x) 2*x.^2 - 8*x - 4 )
>> # Use "hold on" so the first plot isn't erased
>> hold on;
>> ezplot(@(x) 3*x -x.^2 )
>> title ('2*x.^2 - 8*x - 4 = 3*x -x.^2')

Balance Chemical Equations

An example of balancing chemical equations, would be to determine how much methane (CH4 ) and oxygen (O2) would burn cleanly to water(H20) and carbon dioxide (CO2):

CH4 + O2 → CO2 + H2O

To determine the actual amounts of each element variables X1-X4 are added:

x1(CH4) + x2(O2) → x3(CO2) + x4(H2O)

This equation can be broken down by each element:

Carbon (C): 1*x1 + 0*x2 = 1*x3 + 0*x4
Hydrogen (H): 4*x1 + 0*x2 = 0*x3 + 2*x4
Oxygen (O): 0*x1 + 2*x2 = 2*x3 + 1*x4

This equation can be rewritten as:

1*x1 + 0*x2 – 1*x3 + 0*x4 = 0
4*x1 + 0*x2 – 0*x3 – 2*x4 = 0
0*x1 + 2*x2 – 2*x3 – 1*x4 = 0

Now there are 3 equations and 4 variable. To solved these equations, as a first pass x4=1 (now there are 4 equations). The 4 equation can be defined in a matrix equation of : Ax = b:

To solve this in Octave/Matlab the matrices are A and b are defined then the equation is rearranged to:

x = A-1 *b

>> # Balance equation of: Methane + Oxygen -> Carbon Dioxide + Water
>> # Create a Matrix A of elements
>> A = [
1 0 -1 0
4 0 0 -2
0 2 -2 -1
0 0 0 1];

>> # Create a Matric b of results
>> b = [
0
0
0
1];

>> # Ax = b, or x=inverse(A) * b, solve for x
>> x = inv(A) *b
x =

   0.50000
   1.00000
   0.50000
   1.00000

>> # Multiple by 2 to get whole numbers
>> x = 2*x
x =

   1
   2
   1
   2

Originally a guess of x4 = 1 was used, this gave a result with decimal/fractional values. By setting X4 = 2 (multiple results by 2) whole values can returned. The final balanced equation is:

CH4 + 2O2 → CO2 + 2H2O

Dataframes

An Octave dataframe (similar to a Python Pandas dataframe) is used to store rows and columns of data.

Dataframes can be used to filter, sort and do stats on data files.

For my test project I used the data from Wikipedia on the tallest buildings (https://en.wikipedia.org/wiki/List_of_tallest_buildings), and I created a csv file called: buildings.csv .

CSV files can be imported into dataframe objects. I found that for some files the dataframe import failed, as a work around I imported the CSV file first into a csv2cell object.

Below is an example of imported a CSV file and showing the first 6 rows in a bar chart:

>> pkg load dataframe
>> c = csv2cell("buildings.csv");
>> df = dataframe(c);
>> # show the first 3 rows and all the columns
>> df(1:3, : )
ans = dataframe with 3 rows and 6 columns                                
_1   rank           name height theyear         city              country
Nr double           char double  double         char                 char
 1      1   Burj Khalifa   2717    2010        Dubai United Arab Emirates
 2      2    Merdeka 118   2227    2022 Kuala Lumpur             Malaysia
 3      3 Shanghai Tower   2073    2015     Shanghai                China
>>
>> # create a bar chart with the first 6 tallest buildings, show height and name 
>> bar(df(1:6,['height']) )
>> set(gca,'XTickLabel',df(1:6,['name'])

Dataframes can filtered based on conditions with a field. Below is an example of the tallest building in the US that was built after 2014:

>> us = df(strcmp(cellstr(df.country),'United States') & df.theyear > 2014, : )
us = dataframe with 6 rows and 6 columns                                 
_1   rank                 name height theyear          city       country
Nr double                 char double  double          char          char
15     15   Central Park Tower   1550    2020 New York City United States
28     28 111 West 57th Street   1428    2021 New York City United States
29     29       One Vanderbilt   1401    2020 New York City United States
31     31      432 Park Avenue   1397    2015 New York City United States
45     45      30 Hudson Yards   1270    2019 New York City United States
64     64    St. Regis Chicago   1191    2020       Chicago United States

Statistical functions can also be used with filters.

>> # Find the tallest building built up until 2000
>> max(df.height(df.theyear <= 2000))
ans =  1483

Data Sensor Example

Octave has a number of libraries that allow for sensor inputs from Arduino, Raspberry Pi modules and hardware interfaces such as serial, TCP sockets and I2C devices. The data can then be analysed and outputed to SQL databases, CSV files, plots, dialogs and Excel files.

For the last example a script will periodically read a sensor value then:

  • save times/values to a CSV file
  • show the last 20 values in a dynamic plot
  • show a dialog with a moving average of the last 5 values.

The Octave script (sensor1.m) is launched at the command line by: octave sensor1.m .

For this example the sensor value is simulated by using the CPU idle time that is returned from the top utility. The zenity library is loaded and used for the moving average dialog.

There are a number of way to export data to a CSV file. This example uses a simple bash echo with the results piped to sensor1.csv .

#!/usr/bin/octave
#
# sensor1.m - scan a simulated sensor, then graph and save values
#
pkg load zenity

cmd = "top -n 1 | grep %Cpu | awk '{printf $8}'";

p = zenity_progress("Sensor Value ");

# Cycle for 1000 iterations
x = [];
for i = 1:1000
  # get the sensor (idletime) value and time
  thetime = strftime("%X",localtime (time ()));
  [~, idletime] = system(cmd);
  idletime_n = str2num(idletime);
  x(end+1) = idletime_n;
  # Only graph the last 20 values
  if length(x) > 20
    x(1) = [];
  endif
  plot(x)
  title(strcat("Sensor Values   Last Value: ", idletime, "%"))
  # Show a move average of the last 5 points in a dialog
  pause (2);
  if length(x) > 5
    movavg = mean(x(end-5:end));
    the_comment = strcat(idletime, " % (Ave of last 5 samples");
    zenity_progress(p,the_comment,movavg);
  endif
  # append the time and sensor value to a text file
  outcmd = ["echo " thetime " , "  idletime " >> sensor1.csv"];
  system(outcmd);
endfor

Use Raspberry Pi Sensors/Displays

Octave can be loaded on a Raspberry Pi and then I2C, SPI and hardwired sensors and displays can be interfaced to.

Unfortunately Octave doesn’t have the native libraries for these devices but an Octave library called Pythonic can be used to access Python libraries.

The Pythonic library can be used to connected to Pi GPIO or any device that has a Python library. See the documentation for more specifics.

An example to connect to a TM1637 four digit LCD screen with the Octave pyexec commands:

>> pyexec("import tm1637")
>> tm = pyexec("tm1637.TM1637(clk=23, dio=24)")
>> pyexec("tm.temperature(44)")

Below is an example to connect an I2C LCD text display with Python objects mapped to Octave objects. The key is to first use the pyexec command to import the required Python library and then create a Python device object (lcd in this case). Once the Python object is defined the Octave object is by:

Octave_obj = py.Python_object()

octave:5> pyexec("from rpi_lcd import LCD")
octave:6> pyexec("lcd = LCD()")
octave:7> pyexec("lcd.text('Hello Octave', 1)")
octave:8> o = py.LCD()
o = [Python object of type rpi_lcd.LCD]

  <rpi_lcd.LCD object at 0xa316f058>

octave:9> o.clear()
octave:10> o.text('Hi Pete',1)
octave:11> o.text('From Octave',2)

Final Comments

For a high school or university student who needs to do some mathematical or engineer work and doesn’t have access to Matlab, Octave is a great open source alternative.