Monitor Linux Servers with SSH/command line tools and Node-Red

There are a number of technologies and packages available for monitoring computer hardware. For medium to large systems an SNMP (Simple Network Monitoring Protocol) approach is usually the preferred solution. However if you have a smaller system with older servers there are excellent light weight command line utilities that can be used.

These command line utilities can be remotely run using SSH (Secure Shell) and the output is parsed to return only the data value. This value can then be graphically shown in a Node-Red web dashboard.

bash_goals

In this blog I will show some examples using iostat to monitor CPU utilization, and  lm-sensors and hddtemp to monitor temperatures.

 

iostat – CPU Utilitization

The iostat utility is part of the sysstat package and it is probably already loaded on your systems, if not it can be installed by:

 sudo apt-get install sysstat

When iostat will generate a report of CPU, device and file system utilization.

$iostat 
Linux 4.15.0-72-generic (lubuntu) 	2020-04-16 	_i686_	(4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          19.48    0.01    7.96    0.65    0.00   71.90

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
loop0             0.00         0.00         0.00       1543          0
loop1             0.21         0.21         0.00     107980          0
loop2             0.13         0.13         0.00      66224          0
loop3             0.00         0.00         0.00       1141          0
loop4             0.00         0.00         0.00          8          0
sda               1.92        13.08        31.89    6722321   16395304

To find a specific result, an iostat option and some bash code. For example to find just the CPU %idle:

$ iostat -c
Linux 4.15.0-72-generic (lubuntu) 	2020-04-16 	_i686_	(4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          19.45    0.01    7.94    0.65    0.00   71.95

$ # get the 4th line of just stats
$ iostat -c | sed -n 4p
 19.45 0.01 7.94 0.65 0.00 71.95

$  # get the 4th line, 6th string
$ iostat -c | sed -n 4p | awk '{print $6}'
 71.95

lm-sensors – Chip based temperature sensors

To install lm-sensor on Ubuntu enter:

sudo apt-get install lm-sensors

The next step is to detect which sensors are available and need to be monitored:

 sudo sensors-detect

This step will give a number of prompts about which sensor to scan. Once the scan step is complete the sensors command will return results for hardware that it found:

pete@lubuntu:~$  sensors  
dell_smm-virtual-0
Adapter: Virtual device
Processor Fan: 2700 RPM
CPU:            +42.0°C  
Ambient:        +34.0°C  
SODIMM:         +34.0°C  

acpitz-virtual-0
Adapter: Virtual device
temp1:        +48.5°C  (crit = +107.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Package id 0:  +43.0°C  (high = +87.0°C, crit = +105.0°C)
Core 0:        +42.0°C  (high = +87.0°C, crit = +105.0°C)
Core 1:        +40.0°C  (high = +87.0°C, crit = +105.0°C)

Specific sensors can be shown by: sensors the-chip .The grep command can be used to get a specific line in the output. For example to get just the Core 0 temperature:

pete@lubuntu:~$ sensors | grep 'Core 0'
Core 0:        +45.0°C  (high = +87.0°C, crit = +105.0°C)

The awk command can be used to get just the temperature, (the third string on the line).

pete@lubuntu:~$ sensors | grep 'Core 0' | awk '{print $3}'
+45.0°C

Later in Node-Red I will show this value in a graph or chart.

 

hddtemp – Monitor Hard Drive Temperatures

hddtemp is a hard drive temperature monitoring package. It can be installed by:

sudo apt-get install hddtemp

By default hddtemp requires superuser rights, to make the results available to non-superusers enter:

sudo chmod u+s /usr/sbin/hddtemp

To see the temperature of a hard drive, enter the drivers device name. For example to see /dev/sda :

pete@lubuntu:~$ hddtemp /dev/sda
/dev/sda: WDC WD3200BPVT-75JJ5T0: 34°C

Again the awk command can be used to parse the output to get just the temperature. For the /dev/sda example the temperature was the fourth string.

pete@lubuntu:~$ hddtemp /dev/sda | awk '{print $4}'
34°C

Psensor – a Sensors Graphic Widget

This is a little off topic but it is worth mentioning. Psensor offers a widget that will show CPU idle time and it monitors data from lm-sensor and hddtemp.

psensor

Psensor is install by:

sudo apt-get install psensor

Psensor is a slick utility for local monitoring, but it isn’t really designed to pass information to a central monitoring server.

Remotely Running Commands

Rather than having remote Linux servers send data a central node, the central node can periodically poll the remote servers for data.

SSH (Secure Shell) can be used to run remote commands. The only issue is that SSH needs a user to  enter a password. This is fine for manual testing but for automated polling this is a problem. There are a couple of solutions to this problem:

  1. ssh-keygen – can generate an ssh key pair that is stored in a user directory. This allows the standard ssh client to run without entering a password.
  2. sshpass – is an ssh client that includes the username/password as an command line option.

The ssh-keygen approach is recommended for most applications because it does not expose passwords. For the testing I will show the sshpass method and then in the Node-Red project I will use the ssh-keygen approach.

The sshpass is included in many standard distributions and it can be installed by:

sudo apt-get install sshpass

An sshpass example get to get some CPU board and hard drive temperatures would be:

$ sshpass -p pete ssh pete@192.168.0.116 sensors dell_smm-virtual-0
dell_smm-virtual-0
Adapter: Virtual device
Processor Fan: 3044 RPM
CPU: +31.0°C 

$ sshpass -p pete ssh pete@192.168.0.116 hddtemp /dev/sda 
/dev/sda: HTS548040M9AT00: 39°C

Using some grep and awk calls the output can be shortened to just show the temperatures:

$ sshpass -p pete ssh pete@192.168.0.116 sensors | grep temp1 | awk '{print $2}'
+30.5°C

$ sshpass -p pete ssh pete@192.168.0.116 hddtemp /dev/sda | awk '{print $3}'
39°C

With this basic set of commands it is now possible to use Node-Red to periodically poll Linux servers for data.

Node-Red

Node-Red is an web based visual programming environment. Node-Red has a wide variety of nodes that can be wired together to create logic.

NodeRed is pre-installed with the Raspbian images. To install it on other systems see : https://nodered.org/#get-started

For this project I added two nodes:

  • bigssh – a ssh node that saves and uses ssh keygen credentials
  • bigtimer – a timer node, that is used to poll for data

These components can installed either manually, or via the “Manage Palette” menu option.

add_bigssh

A basic test circuit to manually poll a Linux server and return a temperature, would use an injector, a bigssh and a debug node.

ssh_test

The logic to poll a Linux server every minute and put the results on a Node-Red web dashboard would use a bigtimer, a bigssh, a gauge and a chart node.

The bigssh node can pass all those useful parsed commands that we worked on earlier to a remote node. For example the temperature value on the temp1 (on acpitz-virtual-0)  is:

sensors | grep temp1 | awk '{print#2}'

The bigtimer node has a good selection of scheduling and timing function. By default the middle output pin will generate a pulse every minute.

ssh2_dash

After the logic is complete click on the “Deploy” button on the right side of the menu bar. The Node-Red web dashboard is available at: http://node-red:1880/ui/

nr_ui

Final Comments

In this blog I only looked at three command line utilities there are many others that could use this technique.

One thought on “Monitor Linux Servers with SSH/command line tools and Node-Red

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s