Hi. I'm jon.404, a Unix/Linux/Database/Openstack/Kubernetes Administrator, AWS/GCP/Azure Engineer, mathematics enthusiast, and amateur philosopher. This is where I rant about that which upsets me, laugh about that which amuses me, and jabber about that which holds my interest most: *nix.
Smokeping in OpenBSD httpd
OpenBSD maintained a fork of apache httpd 1.3 for some time, but as the fork started to show its age, they dropped it in favor of nginx. A year later, a home-grown secure web server, called simply "httpd" was birthed from the relayd codebase. I just started messing with httpd recently, and I managed to get smokeping running beautifully in it. After a bit, I decided to start gathering latency stats on ipv6, so I added most of that today. I'm pretty happy with the results.
First off, httpd runs chrooted by default. I didn't want to disturb that feature, so I constructed a suitable chroot in /var/www for smokeping. As you can imagine, copying that much perl is...well, not something I would do on purpose without the promise of some reward (such as no longer relying on a LAMP stack to run smokeping). Ahh, progress!
Building a chroot for perl is a bit of a PITA, but it tends to be more tedious then actually difficult. It's one of those tasks that you seriously consider scripting the scripts you wrote to do the heavy lifting once the job is done. No kidding. You can "ldd `which perl`" and start copying, or you can script the copying, and then script the scripts that are copying to pull in other pieces. Nuked a good few hours just figuring out why certain perl modules didn't want to cooperate in chroot.
For the brave among you, please be certain of a few things:
chroot -u www /var/www $path/to/perl $path/to/script
This must *work* before you even monkey around with getting it working within FastCGI within a web server in chroot. Do *not* pass "go" without the above working cleanly. You can thank me later (after the script runs on the command line within chroot).
Once that is complete, you can move forward with configuring httpd. The important thing to remember here is that smokeping runs within fastcgi, and as I previously had fastcgi programs running within httpd, I didn't want to muck everything up, so I added a line such as:
Once that is complete, I had to edit the default rc script to add "-c /var/www -S -s /var/www/run/smokeping.sock" to the start command. This chroots the fcgi daemon to /var/www and creates the socket referenced in httpd.conf.
Next up, configure smokeping. Ahh, smokeping config. Certainly more fun than sendmail's config, but definitely more tedious than mtrg (as long as you aren't graphing CPU/RAM/etc..., of course). I found it easiest to copy the default config to /var/www/etc/smokeping/config, edit it to suit my needs (this will require copying fping and other probe binaries to the chroot!), then launch it from a cron job that performs the chroot. Smokeping will daemonize, and future cronjobs will fail due to the running smokeping. Sounds silly, but if your smokeping daemon dies, cron will fire up a new one for you. Not the most elegant config, but I was in a "sledgehammer" sort of mood when I set that up.
Oh, make certain you remove "/var/www" anywhere it exists in your smokeping config if you chroot smokeping (the daemon, not the fastcgi script).
Now then, you are going to run into problems with permissions. Since I have smokeping (the daemon) running as one user and smokeping (the fastcgi script) running as another user, I had to set the group write flags for my rrd files and a few other pieces. Depending on your needs, running them as the same user may be sufficient.
Once I finished that up, I re-configured smokeping to perform fping6 probing, and started graphing fun stuff (latency to the internet root dns servers, latency between AWS regions, latency on my home internet connection, etc...).
EDIT - If you manage an AWS VPN or DX link, configuring smokeping can really help pin-point where issues reside. For instance, if you have a machine on your local network that pings both VPN endpoint IPs (cleartext, across the internet) and then pings the internal endpoint IPs (the link-local addresses on the AWS side), and then pings an instance within your VPC, you can easily determine where the failure is. Latency spike on the public IP space? Not much you can do unless you have some BGP pull with your provider. Latency on the internal endpoint IPs but not the Internet? That is something to look further into. Latency *just* on the instance, but not via the tunnel? There are a number of reasons this would occur, but at least you can rule out the VPN or DX.
First off, httpd runs chrooted by default. I didn't want to disturb that feature, so I constructed a suitable chroot in /var/www for smokeping. As you can imagine, copying that much perl is...well, not something I would do on purpose without the promise of some reward (such as no longer relying on a LAMP stack to run smokeping). Ahh, progress!
Building a chroot for perl is a bit of a PITA, but it tends to be more tedious then actually difficult. It's one of those tasks that you seriously consider scripting the scripts you wrote to do the heavy lifting once the job is done. No kidding. You can "ldd `which perl`" and start copying, or you can script the copying, and then script the scripts that are copying to pull in other pieces. Nuked a good few hours just figuring out why certain perl modules didn't want to cooperate in chroot.
For the brave among you, please be certain of a few things:
chroot -u www /var/www $path/to/perl $path/to/script
This must *work* before you even monkey around with getting it working within FastCGI within a web server in chroot. Do *not* pass "go" without the above working cleanly. You can thank me later (after the script runs on the command line within chroot).
Once that is complete, you can move forward with configuring httpd. The important thing to remember here is that smokeping runs within fastcgi, and as I previously had fastcgi programs running within httpd, I didn't want to muck everything up, so I added a line such as:
Code:
location "/htdocs/smokeping/smokeping.fcgi*" {
fastcgi socket "/run/smokeping.sock"
root "/"
}
Next up, configure smokeping. Ahh, smokeping config. Certainly more fun than sendmail's config, but definitely more tedious than mtrg (as long as you aren't graphing CPU/RAM/etc..., of course). I found it easiest to copy the default config to /var/www/etc/smokeping/config, edit it to suit my needs (this will require copying fping and other probe binaries to the chroot!), then launch it from a cron job that performs the chroot. Smokeping will daemonize, and future cronjobs will fail due to the running smokeping. Sounds silly, but if your smokeping daemon dies, cron will fire up a new one for you. Not the most elegant config, but I was in a "sledgehammer" sort of mood when I set that up.
Oh, make certain you remove "/var/www" anywhere it exists in your smokeping config if you chroot smokeping (the daemon, not the fastcgi script).
Now then, you are going to run into problems with permissions. Since I have smokeping (the daemon) running as one user and smokeping (the fastcgi script) running as another user, I had to set the group write flags for my rrd files and a few other pieces. Depending on your needs, running them as the same user may be sufficient.
Once I finished that up, I re-configured smokeping to perform fping6 probing, and started graphing fun stuff (latency to the internet root dns servers, latency between AWS regions, latency on my home internet connection, etc...).
EDIT - If you manage an AWS VPN or DX link, configuring smokeping can really help pin-point where issues reside. For instance, if you have a machine on your local network that pings both VPN endpoint IPs (cleartext, across the internet) and then pings the internal endpoint IPs (the link-local addresses on the AWS side), and then pings an instance within your VPC, you can easily determine where the failure is. Latency spike on the public IP space? Not much you can do unless you have some BGP pull with your provider. Latency on the internal endpoint IPs but not the Internet? That is something to look further into. Latency *just* on the instance, but not via the tunnel? There are a number of reasons this would occur, but at least you can rule out the VPN or DX.
Total Comments 0
