WTF!!?? upgradepkg httpd deleted all my websites!

Mig21 · 12-25-2016, 10:06 AM

I thought I got hacked, but it happened immediately after I ran upgradepkg httpd-2.4.25-x86_64-1_slack14.2.txz

After running the command and trying to restart apache I got an error telling me that:

Code:

AH00112: Warning: DocumentRoot [/srv/httpd/htdocs] does not exist
AH00526: Syntax error on line 248 of /etc/httpd/httpd.conf:
DocumentRoot '/srv/httpd/htdocs' is not a directory, or is not readable

Which is weird because /srv/httpd is a logical volume dedicated to apache.

I checked the logs and confirmed that the error started happening only after I did the upgrade. I was confused as hell.

All my websites from /srv/httpd were gone, completely. I had in /drv/httpd/htdocs:

Code:

cgi-bin/  error/  htdocs/  htdocs-cot/  htdocs-dab/  htdocs-dab-redirect/  htdocs-vp/  icons/

And now all I have is:

Code:

root@littlesvr:/srv/httpd# ls -la /srv/httpd/
total 8
drwxr-xr-x 2 root root 4096 Dec 25 10:36 ./
drwxr-xr-x 5 root root 4096 Dec 25 10:36 ../
lrwxrwxrwx 1 root root    8 Dec 25 10:36 www -> /var/www/
root@littlesvr:/srv/httpd# ls -la /srv/httpd/www/
total 24
drwxr-xr-x  6 root root 4096 Dec 23 14:48 ./
drwxr-xr-x 17 root root 4096 Dec 23 14:35 ../
drwxr-xr-x  2 root root 4096 Dec 23 14:48 cgi-bin/
drwxr-xr-x  3 root root 4096 Dec 23 14:48 error/
drwxr-xr-x  3 root root 4096 Dec 25 10:36 htdocs/
drwxr-xr-x  3 root root 4096 Dec 23 14:48 icons/
root@littlesvr:/srv/httpd# ls -la /srv/httpd/www/htdocs/
total 20
drwxr-xr-x  3 root root 4096 Dec 25 10:36 ./
drwxr-xr-x  6 root root 4096 Dec 23 14:48 ../
-rw-r--r--  1 root root   45 Jun 11  2007 index.html
-rw-r--r--  1 root root   45 Jun 11  2007 index.html.bak.5381
drwxr-xr-x 14 root root 4096 Dec 23 14:49 manual/

Which is the default that comes with apache with slackware. All my files are gone!

After scratching my head for a while I looked at doinst.sh in the .txz file and found this in the end:

Code:

( cd srv ; rm -rf httpd )
( cd srv ; ln -sf /var/www httpd )
( cd srv ; rm -rf www )
( cd srv ; ln -sf /var/www www )

Really!? Just like that? It's not funny guys!

Luckily i just realized most of my content is in sql and I don't think I've made any major changes in htdocs since the last full backup. So I'll survive this. But man, this sucks so much.

ponce · 12-25-2016, 11:37 AM

if you look above in the doinst.sh there's also this block

Code:

# Don't wipe out an existing document root:
if [ ! -L srv/www -a -d srv/www ]; then
  mv srv/www srv/www.bak.$$
fi
if [ ! -L srv/httpd -a -d srv/httpd ]; then
  mv srv/httpd srv/httpd.bak.$$
fi

af7567 · 12-25-2016, 11:39 AM

A bit further up in the doinst.sh is

Code:

# Don't wipe out an existing document root:
if [ ! -L srv/www -a -d srv/www ]; then
  mv srv/www srv/www.bak.$$
fi
if [ ! -L srv/httpd -a -d srv/httpd ]; then
  mv srv/httpd srv/httpd.bak.$$
fi

So I guess doinst saw that /srv/httpd was a directory and tried moving it to /srv/httpd.bak.$$. But since your /srv/httpd is a mount point that mv failed and the script continued anyway.

edit:
Just tested mv'ing a mount point which was in use and get

Code:

# mv mp3/ mp3.bak
mv: cannot move 'mp3/' to 'mp3.bak': Device or resource busy

So doinst should check for that failure before continuing.

SCerovec · 12-25-2016, 06:34 PM

I would advise the OP to consider moving the mount-point "upwards" to /srv/ instead of /srv/httpd/ for various reasons this discovered [BUG] included?

not much will be lost, and some will be gained?

Mig21 · 12-25-2016, 06:35 PM

I have other things in srv as well but I'll rename srv/httpd to something different.

bassmadrigal · 12-25-2016, 07:24 PM

What about switching the mount point to /var/www/? By default, /srv/httpd/ is just a symlink to /var/www/ anyway.

Mig21 · 12-25-2016, 07:36 PM

After this experience I will do my best to avoid any standard directories for content. /var/www would have been deleted exactly the same way.

I moved it to /srv/httpd2. Unfortunately /srv/httpd was used in many places in several configuration files, some of which I've never touched before. But again - I'd rather risk breaking apache than risk losing my content.

af7567 · 12-25-2016, 07:40 PM

Quote:

Originally Posted by bassmadrigal

What about switching the mount point to /var/www/? By default, /srv/httpd/ is just a symlink to /var/www/ anyway.

That would be a good workaround, but the bug with trying to mv a mount point should still be fixed. The doinst script obviously attempts to be non-destructive, so that should work properly.

af7567 · 12-25-2016, 07:42 PM

Quote:

Originally Posted by Mig21

After this experience I will do my best to avoid any standard directories for content. /var/www would have been deleted exactly the same way.

I don't think /var/www would have been deleted since upgradepkg should have detected "new" files in there, but any files in the httpd package would have been overwritten.

SCerovec · 12-26-2016, 02:32 AM

Quote:

Originally Posted by Mig21

After this experience I will do my best to avoid any standard directories for content. /var/www would have been deleted exactly the same way.

I moved it to /srv/httpd2. Unfortunately /srv/httpd was used in many places in several configuration files, some of which I've never touched before. But again - I'd rather risk breaking apache than risk losing my content.

This is the other way of solving this right IMO:
Avoid using default paths for data of value- default paths are targets for all sorts of malevolent software.
Or recent experience shows, of bugs also:
Preferably, use paths that, while make sense, cannot be assumed nor guessed?

I try hard to co-locate all my valuable data behind /home/ as ftp is doing, FWIW.
So I usually only backup /home and /etc .

Mig21 · 12-26-2016, 08:05 PM

Quote:

I try hard to co-locate all my valuable data behind /home/ as ftp is doing, FWIW.
So I usually only backup /home and /etc .

I used to do exactly the same thing but after the recent Slackware upgrade I decided to use LVM so that I have more flexibility allocating disk space for various purposes.

Quote:

Avoid using default paths for data of value- default paths are targets for all sorts of malevolent software.
Or recent experience shows, of bugs also

I agree completely.

SCerovec · 12-27-2016, 01:44 AM

An afterthought:
where You anticipating any harm to your data; You could just unmount it before the (any) upgrade?
That could be an strategy too, provided You run the upgrade offline.

The late developments (of events) however seem to encourage people to run servers "virtually":

One is to upgrades one "offline" copy and then just swaps what's running online.

The data (payload) and the server (towing-truck) are kept in opposing realms of "reality":

*- Servers are VMs (trivial to switch back and forth update wise)
*- Data is either (remote) service or partition/share (preferably off the VM host)

The technique is referred as "destiny separation" in some translations.

FWIW.

Mig21 · 12-27-2016, 11:16 AM

Yep, all good points, except in this case if I had my remote data mounted on /srv/httpd - the apache security update would have deleted my remote data too.

Horrible!

SCerovec · 12-27-2016, 03:01 PM

Well it is an unlucky bug, if there are any lucky bugs

bassmadrigal · 12-27-2016, 03:22 PM

Quote:

Originally Posted by Mig21

/var/www would have been deleted exactly the same way.

In looking at the script, I don't think this is the case. It is specifically doing things within the /srv/ folder, so any removals would only occur on that folder or symlink under /srv/. If the data exists in /var/www/, none of that should be touched (well, except for possibly replacing /var/www/htdocs/index.html if the conditions are met).

But, I agree with af7567, this is a bug that should be fixed. Have you considered emailing Pat in case he hasn't noticed this thread?