LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices



Reply
 
Search this Thread
Old 12-24-2009, 05:07 AM   #1
jsalelle
LQ Newbie
 
Registered: Mar 2006
Location: Oliva, País Valencià
Distribution: Ubuntu
Posts: 8

Rep: Reputation: 0
Unhappy Has my linux server been hacked/cracked/attacked?


Hi, this is my first post here in LQ. In the past I've obtained a lot of help reading posts made for others, but now it seems that nobody has a problem similar to mine!

I installed about 5 years ago a small server in my secondary school:
Is a box with Ubuntu 6.06 with 2 SATA HD drives configured as software RAID1. It is the server of the school with the typical services: Samba, Apache, MySQL, NIS, DNS, DHCP and so on.
The machines of the school validate their users against this server via samba (if the client is a windows machine) or via NIS for linux client machines, and the home directory of the users is mapped via samba or via NFS. Also the system has the web server (with moodle), some MySQL databases, etc...
The system has been working without any noticeable problem until last december 22!!
When the system starts it gives a lot of "Segmentation Fault" errors during the second stage boot...
My troubleshooting until now has proved that isn't a hadware problem: I've attached the two HDD to a similar system and the problem was still there the same.
Also it seems to me that isn't a RAID or Hard Disk problem. I checked with mdam and it seemed OK. Now I have a DEGRADED array, but I think that I have made that error during my troubleshooting, because it appeared when I dettached one of the drives for testing.
The first "Segmentation fault" seems to occur when the script loopback is executed on the system start, so I read the line causing the error and it is a "mkdir".
I've done a
Code:
ls -l /bin
and I've noticed that many basic basic program files have their dates changed to the last system starting date and time: bash, cat, mkdir, mknod, dd, df, dir, echo, ln, chgrp, chmod, chown...
The system now ends the booting process, but there aren't any virtual terminals (CTRL+ALT+F1 to F6), the graphical system starts OK and I can validate as an user and navigate thru filesystem with nautilus.
Many services (like networking) don't start (this is my very big problem!!)
In a terminal window if I launch a
Code:
mkdir
(or some other "hacked" program) it gives a Segmentation Fault, but I can create a folder with Nautilus!!!!
Now, I'm really, really lost!!!
Please give me some advice!!
 
Old 12-24-2009, 02:45 PM   #2
HasC
Member
 
Registered: Oct 2009
Location: South America - Paraguay
Distribution: Debian 5 - Slackware 13.1 - Arch - Some others linuxes/*BSDs through KVM and Xen
Posts: 329

Rep: Reputation: 55
before you start thinking on crackers, you should think on HW or SW failures, IMHO

you should remove the computer from the net, and fire up some rescue live CD (like SystemRescue CD, or whatever). And then do some serious checking: memtest, fsck, badblocks, apt-get check... to name a few.
 
Old 12-25-2009, 04:58 AM   #3
jsalelle
LQ Newbie
 
Registered: Mar 2006
Location: Oliva, País Valencià
Distribution: Ubuntu
Posts: 8

Original Poster
Rep: Reputation: 0
Yes, I've already done many of the things you say:

1.- The computer isn't on the net, because one of the first services not working is the network, even the loopback (localhost interface with IP 127.0.0.1) is not working. But also I've disconnected the ethernet cable.

2.- To check against hardware failure I removed the 2 hard drives having all the system and data (are configured as a software RAID1 array) and connected to ANOTHER similar system, and it gives me exactly the same errors and behaviour, so I can confirm it ISN'T a hardware failure!

3.- I'm wondering what a kind of software failure can give the errors I'm having (and, of course, how to proceed to solve them before starting to reinstall a whole new system, which is a VERY long a hard work). The errors are:

a) Many basic commands (mkdir, dd, df, echo, ...) have their creation dates changed to the date and time of the last startup of the computer (it is: the current date), and when some of them are executed (particullary mkdir) it launches a "Segmentation Fault" error. I've checked this on other system, and if you launch a <code>mkdir -l /bin</code> the dates of the files are the date of the installation of the system or the date of the last update.
b) The system is unusable: it doesn't have network and many programs fail to start and no one service is working, BUT the graphical interface (GNOME) is working, so I can check the data are still there, now I'm starting the backup of the web server, the MySql databases, the home directories and many configuration files...
c)I've noticed that GNOME is working but the virtual terminals (CTRL+ALT+F1 to F6) are NOT WORKING. Also I can create a folder with nautilus, in any place of the filesystem if I'm sudoer (if I'm not sudoer I can only create it on my home, of course), but within a terminal if I launch an mkdir command it gives always the Seg Fault error.

So, now I'm stuck, I don't know what to do. I'm starting the work of reinstalling a completely new server and the long work of configuring all the services (particullary I'm affraid of configuring samba as PDC, because I did it long time ago and I'm losing my Windows skills ;-) and passing all the data.

Thank you anyway for your help.
 
Old 12-26-2009, 08:48 AM   #4
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,791
Blog Entries: 1

Rep: Reputation: 414Reputation: 414Reputation: 414Reputation: 414Reputation: 414
Have you combed through the log files at all? Have you looked for processes that shouldn't be there? A few commands you might try to look for things that might be unexpected:

lsof -Pwn
netstat -anpe
ps -axfwwwe

If you still want to do some investigating about the possibility of a crack, the CERT checklist is a pretty good place to start.



I'm a little confused by this one (probably lack of coffee):
Quote:
2.- To check against hardware failure I removed the 2 hard drives having all the system and data (are configured as a software RAID1 array) and connected to ANOTHER similar system, and it gives me exactly the same errors and behaviour, so I can confirm it ISN'T a hardware failure!
Are you saying that the original disks in a new environment behave the same way as they did on the suspect computer? If so, while it does exclude all other hardware, it might point to the disks as the potential hardware failure.

Anyway, looking in the log files would probably be a good place to start.
 
Old 12-27-2009, 03:27 AM   #5
jsalelle
LQ Newbie
 
Registered: Mar 2006
Location: Oliva, País Valencià
Distribution: Ubuntu
Posts: 8

Original Poster
Rep: Reputation: 0
Unhappy

Quote:
Originally Posted by Hangdog42 View Post
Have you combed through the log files at all?
One of the things that are failing during startup is the creation of log files, so I don't have logs for the boot process! I only have dmesg (which don't tell to me anything, I can't see any error) and the logs for mail and other programs and services like this.

Quote:
Originally Posted by Hangdog42 View Post
Have you looked for processes that shouldn't be there? A few commands you might try to look for things that might be unexpected:

lsof -Pwn
netstat -anpe
ps -axfwwwe
I'm already mounting a completely new server, but I'll try these commands later.
Quote:
Originally Posted by Hangdog42 View Post
If you still want to do some investigating about the possibility of a crack, the CERT checklist is a pretty good place to start.
Also i'll try this after the configuration of my new system.
Quote:
Originally Posted by Hangdog42 View Post
...
Are you saying that the original disks in a new environment behave the same way as they did on the suspect computer? If so, while it does exclude all other hardware, it might point to the disks as the potential hardware failure.

...
OK, that's right. If my system has not been attacked it seems to me that the only hardware failure must be in the disks, BUT the two hard drives are mounted as a RAID1 software array, just to protect the system from disk failures!!
In recent checks I've found other strange thing: the /var/www directory can't be seen with nautilus, but it appears in a <code>ls /var</code> so I can do a backup of the web pages of my server!!

Thanks for your replies and help!!
 
Old 12-27-2009, 03:49 AM   #6
DrLove73
Senior Member
 
Registered: Sep 2009
Location: Srbobran, Serbia
Distribution: CentOS 5.5 i386 & x86_64
Posts: 1,118
Blog Entries: 1

Rep: Reputation: 129Reputation: 129
Quote:
Originally Posted by jsalelle View Post
In recent checks I've found other strange thing: the /var/www directory can't be seen with nautilus, but it appears in a <code>ls /var</code> so I can do a backup of the web pages of my server!!
ls is working as root or as a user you use while using nautilus? Maybe there are access issues for regular user?
 
Old 12-27-2009, 09:25 AM   #7
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,791
Blog Entries: 1

Rep: Reputation: 414Reputation: 414Reputation: 414Reputation: 414Reputation: 414
Quote:
One of the things that are failing during startup is the creation of log files, so I don't have logs for the boot process! I only have dmesg (which don't tell to me anything, I can't see any error) and the logs for mail and other programs and services like this.
Sorry, I wasn't too clear. The important log files would be from December 22 or just prior. Certainly current log files would be nice, but if something unusual was going on just before this all started happening it would be in the older files.

The more details you post, the more I think a good solid trip through the CERT checklist is in order. So far there is no evidence to put a crack over hardware failure, but there certainly is a lot of odd goings on.
 
Old 12-27-2009, 02:52 PM   #8
MadAtUbuntu
LQ Newbie
 
Registered: Dec 2009
Location: Northern California USA
Distribution: Ubuntu Server 8.04.3 Hardy
Posts: 14

Rep: Reputation: 0
2.- To check against hardware failure I removed the 2 hard drives having all the system and data (are configured as a software RAID1 array) and connected to ANOTHER similar system, and it gives me exactly the same errors and behaviour, so I can confirm it ISN'T a hardware failure!


That does not confirm that it is not a hardware failure, I suggest that you obtain a copy of the hard disk manufacturers diagnostic tools and illiminate the possibiility of faulty disks first. Putting a bad drive on another system and getting the same results would allude to the possibility of a potential disk failure. As much as that may sound too terrifying to be true, its something you need to face.
 
Old 12-28-2009, 11:55 AM   #9
jantman
Member
 
Registered: Nov 2005
Location: New Jersey, USA
Distribution: SuSE
Posts: 492

Rep: Reputation: 31
Sounds like a failed disk/RAID error to me.

Have you done any updates on this? Do you (oh no) have automatic updates enabled?

This also sounds like it could be from a bad update, leaving some system libraries in an inconsistent state.

Do you have remote syslogs going anywhere? Have you checked them?

Have you tried to compare the MD5sums of some of these binaries with a machine running the same versions?

Any chance you're running a backup system like Bacula that also keeps MD5s of files?

Honestly, I'd say a new install is the best bet.
 
Old 12-30-2009, 09:00 AM   #10
jsalelle
LQ Newbie
 
Registered: Mar 2006
Location: Oliva, País Valencià
Distribution: Ubuntu
Posts: 8

Original Poster
Rep: Reputation: 0
I want to give many thanks to all the people in this forum who has tried to help me!
Thank you very much for the help!
Finally I've installed a completely new server and it seems to me that I haven't lost any data. I can't still test if NIS and Samba as PDC is working well, because the school where the computers clients are is closed, I'll check it next January, 4, I think that all will be going OK!

My Linux skills aren't so deep to check more things to find out what was wrong with the server, also I haven't the time. I think the RAID failed in some way that corrupted the filesystem and the bad data were copied to the two disks (because when the problem happened, mdadm reported the RAID was OK).

Now my server is also with a software RAID1, with Ubuntu LTS 8.04 Server. I installed the core of Gnome in order to have Nautilus, Gedit and Firefox as configuration tools, despite the advices not to install graphical apps in a server, but my skills in the console aren't good enough to work without "grapphic help". But the graphical environment doesn't starts automatically, I have to start it with startx, I hope this helps to get a more secure server!

I repeat: thank ypu very much a lot!!!
 
Old 12-30-2009, 09:37 AM   #11
Jim Bengtson
Member
 
Registered: Feb 2009
Location: Iowa
Distribution: Ubuntu 9.10
Posts: 164

Rep: Reputation: 38
Quote:
My Linux skills aren't so deep to check more things to find out what was wrong with the server, also I haven't the time. I think the RAID failed in some way that corrupted the filesystem and the bad data were copied to the two disks (because when the problem happened, mdadm reported the RAID was OK).
While you're at it, before anything else happens, you might want to install and run Tripwire. Tripwire takes a snapshot of the files on your computer; at any time from then on you can compare the current state of those files to your snapshot, to see what files have changed.
 
Old 12-31-2009, 04:11 AM   #12
DrLove73
Senior Member
 
Registered: Sep 2009
Location: Srbobran, Serbia
Distribution: CentOS 5.5 i386 & x86_64
Posts: 1,118
Blog Entries: 1

Rep: Reputation: 129Reputation: 129
Quote:
Originally Posted by jsalelle View Post
Now my server is also with a software RAID1, with Ubuntu LTS 8.04 Server. I installed the core of Gnome in order to have Nautilus, Gedit and Firefox as configuration tools, despite the advices not to install graphical apps in a server, but my skills in the console aren't good enough to work without "grapphic help". But the graphical environment doesn't starts automatically, I have to start it with startx, I hope this helps to get a more secure server!
I always install full Gnome (basic set that comes with CentOS/RHEL 5.x. If someone is clever enough to brake into the server and gains root access, he can install anything he likes, and is probably better with command line usage anyhow. If he does not have root access, then he will not be able to make any damage.. I add nxserver so I can remotely log-in to Gnome via ssh, and I also install Webmin (web based configuration of entire server (hardware, software, services).

As for the security, I use shorewall firewall, SELinux on RHEL/CentOS is excelent and Tripwire is also good I hear. Make users create good passwords no matter how insignificant access level is, make good access privilegies for folders and files, and half of the job is done.

As for the RAID1, he only makes a copy of the primary data to preserve data in the case of the hardware failure. Any change on the file system is also replicated. For data protection of the data against user or application change use regular backups (tar can be used very easy).
 
  


Reply

Tags
boot, hacked, server


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Help me. My server is attacked DDoS ndduy Linux - Security 12 11-29-2009 03:47 PM
script cracked my server kav Linux - Security 3 08-26-2008 01:08 PM
cracked or not cracked (tripwire & chrootkit) ddaas Linux - Security 1 04-27-2005 08:29 AM
qmail server getting attacked lsimon4180 Linux - Software 41 10-15-2004 04:44 PM
I suspect my linux server is hacked. What should i do ?? td0l2 Linux - Security 6 06-24-2004 05:13 AM


All times are GMT -5. The time now is 01:42 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration