Hi, this is my first post here in LQ. In the past I've obtained a lot of help reading posts made for others, but now it seems that nobody has a problem similar to mine!

I installed about 5 years ago a small server in my secondary school:
Is a box with Ubuntu 6.06 with 2 SATA HD drives configured as software RAID1. It is the server of the school with the typical services: Samba, Apache, MySQL, NIS, DNS, DHCP and so on.
The machines of the school validate their users against this server via samba (if the client is a windows machine) or via NIS for linux client machines, and the home directory of the users is mapped via samba or via NFS. Also the system has the web server (with moodle), some MySQL databases, etc...
The system has been working without any noticeable problem until last december 22!!
When the system starts it gives a lot of "Segmentation Fault" errors during the second stage boot...
My troubleshooting until now has proved that isn't a hadware problem: I've attached the two HDD to a similar system and the problem was still there the same.
Also it seems to me that isn't a RAID or Hard Disk problem. I checked with mdam and it seemed OK. Now I have a DEGRADED array, but I think that I have made that error during my troubleshooting, because it appeared when I dettached one of the drives for testing.
The first "Segmentation fault" seems to occur when the script loopback is executed on the system start, so I read the line causing the error and it is a "mkdir".
I've done a and I've noticed that many basic basic program files have their dates changed to the last system starting date and time: bash, cat, mkdir, mknod, dd, df, dir, echo, ln, chgrp, chmod, chown...
The system now ends the booting process, but there aren't any virtual terminals (CTRL+ALT+F1 to F6), the graphical system starts OK and I can validate as an user and navigate thru filesystem with nautilus.
Many services (like networking) don't start (this is my very big problem!!)
In a terminal window if I launch a (or some other "hacked" program) it gives a Segmentation Fault, but I can create a folder with Nautilus!!!!
Now, I'm really, really lost!!!
Please give me some advice!!

before you start thinking on crackers, you should think on HW or SW failures, IMHO

you should remove the computer from the net, and fire up some rescue live CD (like SystemRescue CD, or whatever). And then do some serious checking: memtest, fsck, badblocks, apt-get check... to name a few.

Yes, I've already done many of the things you say:

1.- The computer isn't on the net, because one of the first services not working is the network, even the loopback (localhost interface with IP 127.0.0.1) is not working. But also I've disconnected the ethernet cable.

2.- To check against hardware failure I removed the 2 hard drives having all the system and data (are configured as a software RAID1 array) and connected to ANOTHER similar system, and it gives me exactly the same errors and behaviour, so I can confirm it ISN'T a hardware failure!

3.- I'm wondering what a kind of software failure can give the errors I'm having (and, of course, how to proceed to solve them before starting to reinstall a whole new system, which is a VERY long a hard work). The errors are:

a) Many basic commands (mkdir, dd, df, echo, ...) have their creation dates changed to the date and time of the last startup of the computer (it is: the current date), and when some of them are executed (particullary mkdir) it launches a "Segmentation Fault" error. I've checked this on other system, and if you launch a <code>mkdir -l /bin</code> the dates of the files are the date of the installation of the system or the date of the last update.
b) The system is unusable: it doesn't have network and many programs fail to start and no one service is working, BUT the graphical interface (GNOME) is working, so I can check the data are still there, now I'm starting the backup of the web server, the MySql databases, the home directories and many configuration files...
c)I've noticed that GNOME is working but the virtual terminals (CTRL+ALT+F1 to F6) are NOT WORKING. Also I can create a folder with nautilus, in any place of the filesystem if I'm sudoer (if I'm not sudoer I can only create it on my home, of course), but within a terminal if I launch an mkdir command it gives always the Seg Fault error.

So, now I'm stuck, I don't know what to do. I'm starting the work of reinstalling a completely new server and the long work of configuring all the services (particullary I'm affraid of configuring samba as PDC, because I did it long time ago and I'm losing my Windows skills ;-) and passing all the data.

Thank you anyway for your help.

Have you combed through the log files at all? Have you looked for processes that shouldn't be there? A few commands you might try to look for things that might be unexpected:

lsof -Pwn
netstat -anpe
ps -axfwwwe

If you still want to do some investigating about the possibility of a crack, the CERT checklist is a pretty good place to start.



I'm a little confused by this one (probably lack of coffee): Are you saying that the original disks in a new environment behave the same way as they did on the suspect computer? If so, while it does exclude all other hardware, it might point to the disks as the potential hardware failure.

Anyway, looking in the log files would probably be a good place to start.

One of the things that are failing during startup is the creation of log files, so I don't have logs for the boot process! I only have dmesg (which don't tell to me anything, I can't see any error) and the logs for mail and other programs and services like this.

I'm already mounting a completely new server, but I'll try these commands later.
Also i'll try this after the configuration of my new system.
OK, that's right. If my system has not been attacked it seems to me that the only hardware failure must be in the disks, BUT the two hard drives are mounted as a RAID1 software array, just to protect the system from disk failures!!
In recent checks I've found other strange thing: the /var/www directory can't be seen with nautilus, but it appears in a <code>ls /var</code> so I can do a backup of the web pages of my server!!

Thanks for your replies and help!!

ls is working as root or as a user you use while using nautilus? Maybe there are access issues for regular user?

Sorry, I wasn't too clear. The important log files would be from December 22 or just prior. Certainly current log files would be nice, but if something unusual was going on just before this all started happening it would be in the older files.

The more details you post, the more I think a good solid trip through the CERT checklist is in order. So far there is no evidence to put a crack over hardware failure, but there certainly is a lot of odd goings on.

2.- To check against hardware failure I removed the 2 hard drives having all the system and data (are configured as a software RAID1 array) and connected to ANOTHER similar system, and it gives me exactly the same errors and behaviour, so I can confirm it ISN'T a hardware failure!


That does not confirm that it is not a hardware failure, I suggest that you obtain a copy of the hard disk manufacturers diagnostic tools and illiminate the possibiility of faulty disks first. Putting a bad drive on another system and getting the same results would allude to the possibility of a potential disk failure. As much as that may sound too terrifying to be true, its something you need to face.

Sounds like a failed disk/RAID error to me.

Have you done any updates on this? Do you (oh no) have automatic updates enabled?

This also sounds like it could be from a bad update, leaving some system libraries in an inconsistent state.

Do you have remote syslogs going anywhere? Have you checked them?

Have you tried to compare the MD5sums of some of these binaries with a machine running the same versions?

Any chance you're running a backup system like Bacula that also keeps MD5s of files?

Honestly, I'd say a new install is the best bet.

I want to give many thanks to all the people in this forum who has tried to help me!
Thank you very much for the help!
Finally I've installed a completely new server and it seems to me that I haven't lost any data. I can't still test if NIS and Samba as PDC is working well, because the school where the computers clients are is closed, I'll check it next January, 4, I think that all will be going OK!

My Linux skills aren't so deep to check more things to find out what was wrong with the server, also I haven't the time. I think the RAID failed in some way that corrupted the filesystem and the bad data were copied to the two disks (because when the problem happened, mdadm reported the RAID was OK).

Now my server is also with a software RAID1, with Ubuntu LTS 8.04 Server. I installed the core of Gnome in order to have Nautilus, Gedit and Firefox as configuration tools, despite the advices not to install graphical apps in a server, but my skills in the console aren't good enough to work without "grapphic help". But the graphical environment doesn't starts automatically, I have to start it with startx, I hope this helps to get a more secure server!

I repeat: thank ypu very much a lot!!!

While you're at it, before anything else happens, you might want to install and run Tripwire. Tripwire takes a snapshot of the files on your computer; at any time from then on you can compare the current state of those files to your snapshot, to see what files have changed.

I always install full Gnome (basic set that comes with CentOS/RHEL 5.x. If someone is clever enough to brake into the server and gains root access, he can install anything he likes, and is probably better with command line usage anyhow. If he does not have root access, then he will not be able to make any damage.. I add nxserver so I can remotely log-in to Gnome via ssh, and I also install Webmin (web based configuration of entire server (hardware, software, services).

As for the security, I use shorewall firewall, SELinux on RHEL/CentOS is excelent and Tripwire is also good I hear. Make users create good passwords no matter how insignificant access level is, make good access privilegies for folders and files, and half of the job is done.

As for the RAID1, he only makes a copy of the primary data to preserve data in the case of the hardware failure. Any change on the file system is also replicated. For data protection of the data against user or application change use regular backups (tar can be used very easy).