There are a number of technologies and packages available for monitoring computer hardware. For medium to large systems an SNMP (Simple Network Monitoring Protocol) approach is usually the preferred solution. However if you have a smaller system with older servers there are excellent light weight command line utilities that can be used.
These command line utilities can be remotely run using SSH (Secure Shell) and the output is parsed to return only the data value. This value can then be graphically shown in a Node-Red web dashboard.
In this blog I will show some examples using iostat to monitor CPU utilization, and lm-sensors and hddtemp to monitor temperatures.
iostat – CPU Utilitization
The iostat utility is part of the sysstat package and it is probably already loaded on your systems, if not it can be installed by:
sudo apt-get install sysstat
When iostat will generate a report of CPU, device and file system utilization.
$iostat
Linux 4.15.0-72-generic (lubuntu) 2020-04-16 _i686_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
19.48 0.01 7.96 0.65 0.00 71.90
Device tps kB_read/s kB_wrtn/s kB_read kB_wrtn
loop0 0.00 0.00 0.00 1543 0
loop1 0.21 0.21 0.00 107980 0
loop2 0.13 0.13 0.00 66224 0
loop3 0.00 0.00 0.00 1141 0
loop4 0.00 0.00 0.00 8 0
sda 1.92 13.08 31.89 6722321 16395304
To find a specific result, an iostat option and some bash code. For example to find just the CPU %idle:
$ iostat -c
Linux 4.15.0-72-generic (lubuntu) 2020-04-16 _i686_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
19.45 0.01 7.94 0.65 0.00 71.95
$ # get the 4th line of just stats
$ iostat -c | sed -n 4p
19.45 0.01 7.94 0.65 0.00 71.95
$ # get the 4th line, 6th string
$ iostat -c | sed -n 4p | awk '{print $6}'
71.95
lm-sensors – Chip based temperature sensors
To install lm-sensor on Ubuntu enter:
sudo apt-get install lm-sensors
The next step is to detect which sensors are available and need to be monitored:
sudo sensors-detect
This step will give a number of prompts about which sensor to scan. Once the scan step is complete the sensors command will return results for hardware that it found:
pete@lubuntu:~$ sensors dell_smm-virtual-0 Adapter: Virtual device Processor Fan: 2700 RPM CPU: +42.0°C Ambient: +34.0°C SODIMM: +34.0°C acpitz-virtual-0 Adapter: Virtual device temp1: +48.5°C (crit = +107.0°C) coretemp-isa-0000 Adapter: ISA adapter Package id 0: +43.0°C (high = +87.0°C, crit = +105.0°C) Core 0: +42.0°C (high = +87.0°C, crit = +105.0°C) Core 1: +40.0°C (high = +87.0°C, crit = +105.0°C)
Specific sensors can be shown by: sensors the-chip .The grep command can be used to get a specific line in the output. For example to get just the Core 0 temperature:
pete@lubuntu:~$ sensors | grep 'Core 0' Core 0: +45.0°C (high = +87.0°C, crit = +105.0°C)
The awk command can be used to get just the temperature, (the third string on the line).
pete@lubuntu:~$ sensors | grep 'Core 0' | awk '{print $3}' +45.0°C
Later in Node-Red I will show this value in a graph or chart.
hddtemp – Monitor Hard Drive Temperatures
hddtemp is a hard drive temperature monitoring package. It can be installed by:
sudo apt-get install hddtemp
By default hddtemp requires superuser rights, to make the results available to non-superusers enter:
sudo chmod u+s /usr/sbin/hddtemp
To see the temperature of a hard drive, enter the drivers device name. For example to see /dev/sda :
pete@lubuntu:~$ hddtemp /dev/sda /dev/sda: WDC WD3200BPVT-75JJ5T0: 34°C
Again the awk command can be used to parse the output to get just the temperature. For the /dev/sda example the temperature was the fourth string.
pete@lubuntu:~$ hddtemp /dev/sda | awk '{print $4}' 34°C
Psensor – a Sensors Graphic Widget
This is a little off topic but it is worth mentioning. Psensor offers a widget that will show CPU idle time and it monitors data from lm-sensor and hddtemp.
Psensor is install by:
sudo apt-get install psensor
Psensor is a slick utility for local monitoring, but it isn’t really designed to pass information to a central monitoring server.
Remotely Running Commands
Rather than having remote Linux servers send data a central node, the central node can periodically poll the remote servers for data.
SSH (Secure Shell) can be used to run remote commands. The only issue is that SSH needs a user to enter a password. This is fine for manual testing but for automated polling this is a problem. There are a couple of solutions to this problem:
- ssh-keygen – can generate an ssh key pair that is stored in a user directory. This allows the standard ssh client to run without entering a password.
- sshpass – is an ssh client that includes the username/password as an command line option.
The ssh-keygen approach is recommended for most applications because it does not expose passwords. For the testing I will show the sshpass method and then in the Node-Red project I will use the ssh-keygen approach.
The sshpass is included in many standard distributions and it can be installed by:
sudo apt-get install sshpass
An sshpass example get to get some CPU board and hard drive temperatures would be:
$ sshpass -p pete ssh pete@192.168.0.116 sensors dell_smm-virtual-0 dell_smm-virtual-0 Adapter: Virtual device Processor Fan: 3044 RPM CPU: +31.0°C $ sshpass -p pete ssh pete@192.168.0.116 hddtemp /dev/sda /dev/sda: HTS548040M9AT00: 39°C
Using some grep and awk calls the output can be shortened to just show the temperatures:
$ sshpass -p pete ssh pete@192.168.0.116 sensors | grep temp1 | awk '{print $2}' +30.5°C $ sshpass -p pete ssh pete@192.168.0.116 hddtemp /dev/sda | awk '{print $3}' 39°C
With this basic set of commands it is now possible to use Node-Red to periodically poll Linux servers for data.
Node-Red
Node-Red is an web based visual programming environment. Node-Red has a wide variety of nodes that can be wired together to create logic.
NodeRed is pre-installed with the Raspbian images. To install it on other systems see : https://nodered.org/#get-started
For this project I added two nodes:
- bigssh – a ssh node that saves and uses ssh keygen credentials
- bigtimer – a timer node, that is used to poll for data
These components can installed either manually, or via the “Manage Palette” menu option.
A basic test circuit to manually poll a Linux server and return a temperature, would use an injector, a bigssh and a debug node.
The logic to poll a Linux server every minute and put the results on a Node-Red web dashboard would use a bigtimer, a bigssh, a gauge and a chart node.
The bigssh node can pass all those useful parsed commands that we worked on earlier to a remote node. For example the temperature value on the temp1 (on acpitz-virtual-0) is:
sensors | grep temp1 | awk '{print#2}'
The bigtimer node has a good selection of scheduling and timing function. By default the middle output pin will generate a pulse every minute.
After the logic is complete click on the “Deploy” button on the right side of the menu bar. The Node-Red web dashboard is available at: http://node-red:1880/ui/
Final Comments
In this blog I only looked at three command line utilities there are many others that could use this technique.
Fun Tech Projects 
One thought on “Monitor Linux Servers with SSH/command line tools and Node-Red”