Smokeping
Smokeping allows you to probe a list of servers, store that data using RRDtool, and generate statistical charts based on RRDtool's output. Smokeping consists of two parts. A daemon runs in the background pinging and collecting data at set intervals. A web interface displays that data in the form of graphs.
This guide only sets up the smokeping daemon and CGI script. We do not set up advanced features like the AJAX traceroute or the JavaScript graph cropper. Why? Because these did not work properly for me and I do not need these features enough to figure out why.
Contents
Installation
The smokeping CGI script runs under SpeedyCGI. SpeedyCGI is a modified version of perl that allows scripts to be run persistently. Without persistence, the smokeping CGI script must load all modules and data on every webpage view. This would be too slow.
SpeedyCGI has not been updated since 2003 (as of Nov 1, 2010) and no
longer compiles with perl, which has since gone through API changes.
Instead of using SpeedyCGI for persistence, we will convert the
smokeping.cgi
script into a
PSGI webapp
which can then be made persistent a variety of ways. More on that
later.
We will just be installing the smokeping chart interface as a webapp. The traceroute feature will not be included.
Prerequisites
- perl
- The smokeping daemon and cgi script are perl scripts. We will use the system-wide perl for this example. You might also consider a perl for only the smokeping app. You may also compile an old version of perl (try 5.8.x) if you choose to use SpeedyCGI.
- fping
- Smokeping has many many different ways to probe a host. The simplest and default method is a simple ping probe which requires the fping program.
- rrdtool
- RRDTool is created by the same author as smokeping. RRDTool collects and displays statistical information in a disk-efficient manner.
- perl-plack
- Plack is perl's version of Ruby's rack. Plack is the implementation of the PSGI specification. PSGI is like Python's WSGI. By using Plack/PSGI we can run the smokeping.cgi as a webapp on any number of backends (proxied, FreeCGI, mod_perl, CGI, etc).
- perl-cgi-emulate-psgi
- The CGI::Emulate::PSGI fools the smokeping.cgi script into thinking it is running in a CGI environment, when instead it is executing inside a PSGI app.
- ttf-dejavu
- Without this font the graphs are unreadable and the letters are replaced with blocks.
Optional Prerequisites
If you want to use other probes such as the DNS or http probe you will need other packages shown below. We will only be setting up the basic ping probe in this example.
Probe | Package Needed |
---|---|
Curl | curl |
DNS | dnsutils (for the dig utility) |
EchoPing | echoping |
SSH | openssh |
TelnetIOSPing | perl-net-telnet |
AnotherDNS | perl-net-dns |
LDAP | perl-ldap |
LDAP (tls) | perl-io-socket-ssl |
Authen | perl-authen-radius |
In our example we use a simple Plack-based webserver. You may wish to install a full-fledged webserver. If you wish to use the master/slave functionality of Smokeping you will need the perl-libwww package. If you want to probe IPv6 servers you will need the perl-socket6 package from the AUR. I will not be covering any of these features and will stick to the bare minimum.
Installing Packages
First install the packages that are available from the official repositories:
sudo pacman -S rrdtool fping perl ttf-dejavu
The perl-cgi-emulate-psgi and perl-plack prerequisites can be installed from the AUR.
Downloading Smokeping
We are going to download Smokeping and extract it in our home directory so we can edit it.
cd curl -O http://oss.oetiker.ch/smokeping/pub/smokeping-2.4.2.tar.gz tar zxf smokeping-2.4.2.tar.gz
Configuration
Smokeping requires you to edit a few files. The unedited files end
with the .dist
extension. Rename the .dist
files to names without the suffix. The find command does this and
prints out each file that is being renamed and needs editing:
cd smokeping-2.4.2 find . -name '*.dist' -print -execdir sh -c 'mv {} $(basename {} .dist)' \;
Editing Scripts
The files inside the bin/ directory I call scripts. The script files
contain use lib
statements. These add a path to search
for when loading perl modules (similar to $PATH
). We do not
need these. We will add to the module search path at runtime.
You must set the proper config file path. The config file is by
default set to etc/config.dist
. We will remove the
.dist
suffix. I prefer relative directories so that I
can move the smokeping directory to anywhere I please without re-editing
these files. The downside to using relative paths is you must be sure
to start the daemon and webapp in the proper directory.
Run this sed command to edit these scripts:
sed -i -e '/^use lib/d' -e 's/config.dist/config/' bin/*
Editing the config file
Next edit the etc/config
file. The first section of the
etc/config
file is the easiest to edit. Personalize the
top of the config file to match your information. Replace the absolute
paths to relative paths. The General section should look similar
to my example below. The comments describe each field.
config
*** General *** owner = Your Name Here # your name contact = your.email@host.bla # your email mailhost = your.smtp.server.bla # your mail server sendmail = /usr/sbin/sendmail # where the sendmail program is imgcache = public/images # filesystem directory where we store files imgurl = images # URL directory to find them datadir = var/pingdata # where we share data between the daemon and webapp piddir = var # filesystem directory to store PID file cgiurl = http://yourdomain/smokeping # exterior URL smokemail = etc/smokemail # new relative dir, remove .dist tmail = etc/tmail # ditto syslogfacility = local0 # each probe is now run in its own process # disable this to revert to the old behaviour # concurrentprobes = no
We are setting up the bare minimum and are not using alerts, so comment out
the *** Alerts ***
section. Scroll down to the *** Presentation ***
section and replace the template path with a relative path:
config
*** Presentation *** template = etc/basepage.html
The *** Probes ***
section specify which probes are
active. By default only the FPing probe is set. Leave this for
now. Comment out or delete the *** Slaves ***
section,
you do not have any slaves... right? Scroll down to the
*** Targets ***
section. This is where you specify which
hosts to ping. Replace it with a host you would like to collect statistics on,
like so:
config
*** Targets *** probe = FPing menu = Top title = Network Latency Grapher remark = Hello World! + ArchLinux menu = ArchLinux title = ArchLinux Hosts host = /ArchLinux/ArchLinux_News /ArchLinux/ArchLinux_BBS /ArchLinux/ArchLinux_Wiki ++ ArchLinux_News menu = News title = ArchLinux Main Site host = archlinux.org ++ ArchLinux_BBS menu = BBS title = ArchLinux BBS host = bbs.archlinux.org ++ ArchLinux_Wiki menu = Wiki title = ArchLinux Wiki host = wiki.archlinux.org
Each + character defines a section in our hierarchy. Spaces are
not allowed in the section names. You can define host
as
either a real host name or the path to another section to generate a
multiple host chart. These hosts are all on the same machine so the
statistics are not very interesting.
You can learn more about the Smokeping config file with the examples at http://oss.oetiker.ch/smokeping/doc/smokeping_examples.en.html
For now let us just try to get things started.
Setup
Create the directories we specified or else smokeping will error out.
mkdir -p var/pingdata public/images
Now we create the PSGI app that will wrap the CGI script. If you open the htdocs/smokeping.cgi file you will see it is really simple and just calls the Smokeping module's cgi subroutine. We are going to fool the CGI code into thinking that it is using the CGI environment. Here is where Plack comes into play.
Inside the smokeping dir, create a new file called
app.psgi
. Copy/paste the following into the file:
app.psgi
# -*- cperl -*- use Plack::App::Directory; use Plack::Builder; use CGI::Emulate::PSGI; use CGI qw(); use warnings; use strict; use Smokeping 2.004002; my $smokeping = sub { CGI::initialize_globals(); Smokeping::cgi("etc/config"); }; my $fileapp = Plack::App::Directory->new ({ 'root' => 'public' })->to_app; $smokeping = CGI::Emulate::PSGI->handler( $smokeping ); builder { mount "/" => $smokeping; mount "/smokeping/" => $fileapp; };
This file does the same thing as smokeping.cgi
. We are in
effect fooling smokeping to think it is still run with CGI. The
difference is we wrap the CGI app into a PSGI app. This allows it to
run persistently among other things. The concept of SpeedyCGI is
similar. We also add file serving for the static files like images. The
default, empty, or root path is set to the smokeping app itself and
anything under the smokeping/
URL component serves a
file from the public/
directory. The only files we
serve are the graph images files.
Testing
Now we are going to test our webapp. Type the following commands to start the smokeping daemon and a webserver:
perl -Ilib bin/smokeping plackup -Ilib
Here the -I switch adds the lib
directory to perl's
module search path. This does the same thing as use lib
;
Our commands start the smokeping daemon and a simple server for the webapp which listens on port 5000 by default. Go to the webpage (yourdomain.com:5000) and look at the pretty graph. Keep in mind the default setup pings every 5 minutes so you may have to wait for some data to appear. If something goes wrong, fix it before continuing.
If it works and that is all you wanted feel free to stop here. You can run your webapp with a better server by installing perl-starman from the AUR. Starman is a fast production quality pre-forking server. I usually set it to a very low number of worker processes though because I do not receive a lot of traffic.
You should set the plack environment to deployment if you do not want/need debugging information like stack traces printed. By default Plack runs in the development environment. For example:
starman -Ilib -p 80 -E deployment --workers 2 &
The -I switch does the same thing as PERL5LIB. -p listens on port 80, the default http port. -E sets the plack environment to deployment which disables bug checking.
Deployment
Now you should probably setup a user for smokeping and
create a /etc/rc.d/
daemon startup script to start the
smokeping daemon as well as the FastCGI webapp daemon. You could also
create a directory under /srv/http
for smokeping. When
you copy the files, you only need the following files and directories
and you can leave the rest.
- app.psgi
- lib/
- etc/
- var/
- public/
Copy those files and directories into the destination. The rest is junk! Make sure to cd into the destination before starting the smokeping daemon and FastCGI webapp.
You should also setup the webserver to serve the files and remove file
serving code from from app.psgi
so it looks like this:
app.psgi
# -*- cperl -*- use CGI::Emulate::PSGI; use CGI qw(); use warnings; use strict; use Smokeping 2.004002; my $smokeping = sub { CGI::initialize_globals(); Smokeping::cgi("etc/config"); }; CGI::Emulate::PSGI->handler( $smokeping );
Much simpler!
Proxied
You can continue to use a plack web server. If you do please upgrade from plackup which is designed for testing purposes. The main web server will act as a proxy and forward requests to the plack server.
nginx
I use nginx so here is a quick walkthrough. I like this method because it is quick and easy. Make sure that the port that the Plack server is running on is blocked from outside by a proxy (like iptables). Edit your nginx configuration file similar to this:
nginx.conf
location /smokeping { error_log /srv/http/smokeping/var/error.log info; access_log /srv/http/smokeping/var/access.log combined; proxy_buffers 8 16k; proxy_buffer_size 32k; proxy_pass http://localhost:9003/; } location /smokeping/ { error_log /srv/http/smokeping/var/error.log info; access_log /srv/http/smokeping/var/access.log combined; alias /srv/http/smokeping/public/; }
Nginx will serve our image files for us and will pass any request for /smokeping to the proxied webapp server. When you start the proxied server, make sure it has the port set:
cd /srv/http/smokeping starman -Ilib -p 9003 -E deployment --workers 2 &
Feel free to use any port you like instead of 9003.
FastCGI
You can easily set up the webapp to use FastCGI. Just use plackup to start the FastCGI server process:
plackup -s FCGI --listen /var/run/smokeping.sock --daemonize --nproc 4
(You will need the perl FCGI module installed via the 'perl-fcgi' package)
Now just setup your web server (i.e. apache, lighttpd, nginx) to
route traffic to your FastCGI "server" by pointing to the
/var/run/smokeping.sock
socket.
Advanced Configuration
There are many idiosyncracies to the smokeping configuration file. There is also alot of power. You can setup many different types of probes. You can setup slave smokeping servers that can send their statistics and show you probes from other servers. You can also create your custom probes in perl. I have not gotten that far along and the example configuration file above should be a good start.
Idiosyncracies
I have found some things to avoid in config files through trial and error. Make sure you do not use spaces or periods in section names. You should probably avoid forward slashes too. Sections are defined with +s.
This is probably because the RRD files are stored under the data directory with the same exact names as the sections.
Bugs
There are a few bugs I have noticed.
Tr.cgi
The traceroute CGI script simply does not work. This could be a mistake on my part or a bug in the code. I do not need it or like it so I simply do not use it.
Cropper
I had noticed that if I try to use the javascript cropper it does not work very well. Click on a graph long enough and it will pop up. If I try to crop a graph to zoom in, the server crashes. Hard.
I decided to remove the cropper until I could track down the bug. To
do this remove the <script>
tags inside the
etc/basepage.html
file. You may have noticed that I
created a new file serving dir called public/
. This
avoids all the old files I do not use that are needed for the cropper
and the traceroute script. This is not the most ideal fix and the
server is likely still vulnerable to crashes but at least they will not
happen on accident.
CPU Crunch
Every once and awhile I noticed that the CPU usage goes crazy for the web interface. When I use Twiggy the CPU usage goes up to 100% and stays there, locking up the server. I have decided instead to use the Starman server. That way if one process goes to crap Starman will spawn another one. This could be caused by a race condition in the Smokeping module. I have not really tried to track it down.
Installation with FastCGI in Apache
If we do not need to "trick" Smokeping as described above and run it with Apache/FastCGI we follow the following steps.
1. We follow the "Prerequisites" and "Installing Packages" as described above. We do not want PSGI so we can skip the PSGI related parts. Instead we setup FastCGI as described in [1]:
pacman -S mod_fastcgi
We edit /etc/httpd/conf/httpd.conf so it includes
LoadModule fastcgi_module modules/mod_fastcgi.so <IfModule fastcgi_module> AddHandler fastcgi-script .fcgi # you can put whatever extension you want </IfModule> ScriptAlias /smokeping/smokeping.cgi /opt/smokeping-2.6.9/htdocs/smokeping.fcgi.dist Alias /smokeping /opt/smokeping-2.6.9/htdocs <Directory "/opt/smokeping-2.6.9/htdocs"> Options FollowSymLinks AllowOverride all Order Deny,Allow Allow from all Require all granted </Directory>
2. We download, configure, and install smokeping (at the moment trunk is 2.6.9). This creates /opt/smokeping-2.6.9
. In that folder we will store everything smokeping related files, like the .rrd files, and the image cache. We can change the latter, by editing etc/config
.
cd curl -O http://oss.oetiker.ch/smokeping/pub/trunk/smokeping-trunk.tar.gz tar xf smokeping-trunk.tar.gz cd smokeping-2.6.9 ./configure sudo make install
3. We remove .dist
from the installed smokeping
folder:
cd /opt/smokeping-2.6.9 find . -name '*.dist' -print -execdir sh -c 'mv {} $(basename {} .dist)' \;
4. We edit the config file as described above, so the final file (e.g. /opt/smokeping-2.6.9/etc/config
could look like this:
/opt/smokeping-2.6.9/etc/config
*** General *** owner = Your Name Here # your name contact = your.email@host.bla # your email mailhost = your.smtp.server.bla # your mail server sendmail = /bin/false # where the sendmail program is imgcache = /opt/smokeping-2.6.9/cache # filesystem directory where we store files imgurl = cache # URL directory to find them datadir = /opt/smokeping-2.6.9/data # where we share data between the daemon and webapp piddir = /opt/smokeping-2.6.9/var # filesystem directory to store PID file cgiurl = http://localhost/smokeping/smokeping.cgi # exterior URL smokemail = /opt/smokeping-2.6.9/etc/smokemail tmail = /opt/smokeping-2.6.9/etc/tmail syslogfacility = local0 # each probe is now run in its own process # disable this to revert to the old behaviour # concurrentprobes = no *** Database *** step = 300 pings = 20 # consfn mrhb steps total AVERAGE 0.5 1 1008 AVERAGE 0.5 12 4320 MIN 0.5 12 4320 MAX 0.5 12 4320 AVERAGE 0.5 144 720 MAX 0.5 144 720 MIN 0.5 144 720 *** Presentation *** template = /opt/smokeping-2.6.9/etc/basepage.html + charts menu = Charts title = The most interesting destinations ++ stddev sorter = StdDev(entries=>4) title = Top Standard Deviation menu = Std Deviation format = Standard Deviation %f ++ max sorter = Max(entries=>5) title = Top Max Roundtrip Time menu = by Max format = Max Roundtrip Time %f seconds ++ loss sorter = Loss(entries=>5) title = Top Packet Loss menu = Loss format = Packets Lost %f ++ median sorter = Median(entries=>5) title = Top Median Roundtrip Time menu = by Median format = Median RTT %f seconds + overview width = 600 height = 50 range = 10h + detail width = 600 height = 200 unison_tolerance = 2 "Last 3 Hours" 3h "Last 30 Hours" 30h "Last 10 Days" 10d "Last 400 Days" 400d *** Probes *** + FPing binary = /usr/sbin/fping *** Targets *** probe = FPing menu = Top title = Network Latency Grapher remark = Welcome to the SmokePing website of Arch User. \ Here you will learn all about the latency of our network. + targets menu = Targets ++ Server menu = Arch Linux title = Arch Linux Website host = 66.211.214.131 ++ MultiHost menu = Multihost title = Arch Wiki and Kernel.org host = 5.9.250.164 149.20.4.69
5. We setup smokeping into Apache:
ln -s /opt/smokeping-2.6.9/cache /opt/smokeping-2.6.9/htdocs/cache chown -R http /opt/smokeping-2.6.9/cache chown -R http /opt/smokeping-2.6.9/data
6. Start smokeping and Apache:
/opt/smokeping-2.6.9/bin/smokeping systemctl start http
7. Wait a few minutes and check the first data at http://localhost/smokeping/smokeping.cgi