LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 12-17-2003, 11:38 PM   #1
BxBoy
Member
 
Registered: Oct 2002
Location: New York City (nYc)
Distribution: Red Hat Enterprise 3
Posts: 93

Rep: Reputation: 15
Help, system is rebooting nightly..


For the past week or so, I've noticed that the system has rebooted quite frequently. Before this, it never rebooted. I would like to diagnose the problem, but don't know where to look. I've tried the /var/log/message file, but it hasn't been much help.

What else can I do? I hope the system hasn't been compromised or anything.

Here's a log of the 'last' events..

Quote:
root pts/0 111.222.333.444 Thu Dec 18 00:37 still logged in
reboot system boot 2.4.18-26.8.0 Wed Dec 17 23:28 (01:21)
root pts/0 111.222.333.444 Wed Dec 17 12:30 - 13:15 (00:45)
reboot system boot 2.4.18-26.8.0 Wed Dec 17 04:10 (20:39)
root pts/0 111.222.333.444 Tue Dec 16 16:01 - 16:14 (00:13)
reboot system boot 2.4.18-26.8.0 Tue Dec 16 09:58 (1+14:51)
reboot system boot 2.4.18-26.8.0 Tue Dec 16 06:46 (1+18:03)
reboot system boot 2.4.18-26.8.0 Tue Dec 16 06:24 (1+18:25)
reboot system boot 2.4.18-26.8.0 Tue Dec 16 05:59 (1+18:50)
reboot system boot 2.4.18-26.8.0 Tue Dec 16 04:09 (1+20:40)
root pts/0 111.222.333.444 Sat Dec 13 19:40 - 19:44 (00:03)
root pts/0 111.222.333.444 Sat Dec 13 19:39 - 19:40 (00:00)
reboot system boot 2.4.18-26.8.0 Sat Dec 13 09:36 (4+15:13)
root pts/0 134.74.74.67 Fri Dec 12 16:42 - 17:34 (00:52)
root pts/0 111.222.333.444 Mon Dec 8 23:35 - 23:44 (00:08)
root pts/0 111.222.333.444 Sun Dec 7 13:10 - 15:19 (02:08)
root pts/0 111.222.333.444 Sun Dec 7 02:32 - 02:44 (00:12)
root pts/0 111.222.333.444 Sun Dec 7 02:27 - 02:30 (00:02)

real ip replaced w/ 111.222.333.444
 
Old 12-18-2003, 04:53 AM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Could you tell us a bit more:
- About the box distribution and release?
- If the box has all the updates for the release?
- About the box purpose (daemons running) and if it's on a LAN or remote?
- How many people (are supposed to) have accounts on the box and if they have root access?
- What you|they use to log in (telnet, Ssh etc etc)?
- If you can at least verify the root login times shown are by you? (And why on earth it only shows root logins?!)
- If you're the admin for this box.
- If it's running a(ny form of) watchdog?
- If it's running a firewall and if it's behind a firewall?
- Any IDS like Snort, or other detection capabilities like Scandetd, Ippl, Psad, Portsentry or similar?
- Any other details you think we should know like running or not running the brk patch, and/or running any (recently) found vulnerable software rsync or cvs etc etc.


The times don't follow a pattern I can see, and since you wrote "What else can I do? I hope the system hasn't been compromised or anything." I take it you have doubts. When you have doubts about a box, you need to make certain it isn't compromised, prepare to check the integrity of whole system. If you're not the admin for the box, first alert them and let them handle the case. If the subject is a remote box, then please take precautions. It would be best to use another box, one you can definately trust, to check and compile stuff on before moving it to the remote one, and to retrieve logs and reports back on.
Also note that if a box is actually compromised, any results may not match up with reality and any activity may be logged with a sniffer to catch passwords. If one runs a rootkit, it's easy to hide their presence and change the output of ls, ps, du, df, ifconfig etc etc by replacing binaries or "rerouting" system calls in the kernel. This all may sound grave and threathening, but you better know about these things.


When I say "check out" below, it means you should look for any messages (or absence of) out of the ordinary. You could also tarball all of them and move them to your workstation and inspect there. in that case it would be easy to also add /etc, /var/lib/rpm and /var/log. If you like "clean" output (as opposed to reading the "script" session log) you can append this pipe: "2>&1|tee -a /tmp/output.log" to any command you run and have all output logged into one file.

- Note logging in may be sniffed and/or watched. If there's a cracker on the box, she may be provoked to step up malicious activity and for instance deny access or delete the whole system. This is a risk you may or may not be willing to take. If you're not logged in, log in over Ssh to an unprivileged account and start up a "script" session. Take the time to read this users shell history and crontab. Su(do) to root. Read root's shell history, crontab and /etc/crontab and /etc/cron.* files for known atd and crond scripts. Check other users crontabs and users shell history files.
- Running "grep -v ^# /etc/syslog.conf |grep /" gives you the system logs to check out, and "chkconfig --list" the daemons it knows about. Check out their configs for logs, then check out all logs for error messages.
- If you have the binary, check with "/sbin/ip link show" if any interface is promiscuous mode. Else "ifconfig -a" should do, but it's not 100% dependable because it doesn't pick up everything. An interface running in promiscuous mode (if you didn't do that yourself) is a good indication of possible malicious activity.
- If you run a filesystem integrity checker like Aide, Samhain or tripwire, and you have backed up the databases (preferably to read-only media), use those databases to run the check with. Check the report for changes in checksum, inode, modif/access/change time, size, and check for new files you can't account for or don't know the purpose of.
- Check which version the (remote) rpm binary is and check its md5sum with a known clean one, from an rpm from cdrom or ftp. If the md5sum matches(, and you have backed up the databases, use those databases to run the check with and) run "rpm -Va" 2>&1|tee -a /tmp/output.log". This will give you the results on screen and log to a logfile. Check the logfile for changes in md5sums, times and size you cannot account for. If you ran a filesystem integrity check, "diff" the reports so you can check out files that have attributes that do not match between reports or files that are not in the rpm database.
- Compile Chkrootkit, move it over and run. Check that report.
- Make the backup tarball of /etc, /var/lib/rpm, /var/log, and all the reports you have gathered. Log out of root, Log out of the unprivileged users "script" shell, then copy the tarball over and exit your session.

This isn't complete, but it should get you started. Please remember the more concise you answer the questions and the more detailed you reply after reading all log output, the "better" and faster we can try to assess if the box meets base requirements (in the sense it could be trusted) and work towards resolving the problem of spurious reboots.
 
Old 12-18-2003, 12:25 PM   #3
BxBoy
Member
 
Registered: Oct 2002
Location: New York City (nYc)
Distribution: Red Hat Enterprise 3
Posts: 93

Original Poster
Rep: Reputation: 15
Quote:
Originally posted by unSpawn
Could you tell us a bit more:
- About the box distribution and release?

RedHat 9.0

- If the box has all the updates for the release?

I had it setup to check the RHN network for updates

- About the box purpose (daemons running) and if it's on a LAN or remote?

Basically a web server, it's hosting a webmail client and a forum board. Daemons running include: mysqld, httpd (apache-2.0), sshd. FTP, telnet, are turned off.

- How many people (are supposed to) have accounts on the box and if they have root access?

Two People. But I'm the sole admin.

- What you|they use to log in (telnet, Ssh etc etc)?

Only SSH. telnet is disabled

- If you can at least verify the root login times shown are by you? (And why on earth it only shows root logins?!)

Because I need 'root' access to do maintence. It's a server box, hence I don't need to login as users.

- If you're the admin for this box.

Yes

- If it's running a(ny form of) watchdog?

No.

- If it's running a firewall and if it's behind a firewall?

The default firewall is setup, and it's set to high (I allowed the ssh and httpd access to get thru). It's also behind a corporate CISCO PIX firewall (that blocks everything, but only allows selective ports).

- Any IDS like Snort, or other detection capabilities like Scandetd, Ippl, Psad, Portsentry or similar?

No, but I will think about setting one up. Which would you recommend?

- Any other details you think we should know like running or not running the brk patch, and/or running any (recently) found vulnerable software rsync or cvs etc etc.

Not that I know of. Everything seem'ed to be within the regular parameters.
To be Honest, I threw the compromised thing as a possibility, but I'm very doubtful. I think it's about 5% chance. I checked the network connections with 'netstat -an' and have noticed any unusual connections to the box. It might just be a software conflict somewhere that is causing the system to reboot. But I would like to explore all possibilities before rooting anything out.

Hopefully I provided enough info to get things going, if not, please ask. Thanks for the reply.

Edit: I'd also like to add that it just started happening the past couple of days. So I should probably monitor it a little more (within the next few days) to see if I notice anything out of the ordinary.

Last edited by BxBoy; 12-18-2003 at 12:30 PM.
 
Old 12-18-2003, 03:21 PM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
First of all, thanks for replying with those details, it really helps to get a better idea of your setup.

Because I need 'root' access to do maintence. It's a server box, hence I don't need to login as users.
It really would be wise to make it SOP to log in as unprivileged user, then use sudo or su to root.


- Any IDS like Snort, or other detection capabilities like Scandetd, Ippl, Psad, Portsentry or similar?
No, but I will think about setting one up. Which would you recommend?

Snort, definately. And since you're behind the PIX and already stopped running a lot of services you don't have to load all rulesets.


- Any other details you think we should know like running or not running the brk patch, and/or running any (recently) found vulnerable software rsync or cvs etc etc.
Not that I know of. Everything seem'ed to be within the regular parameters.

Then I hope your 2.4.18-26.8.0 kernel is the latest update with the backported brk patch.
If it ain't, it isn't hard to compile it as module and load it. Not that this vulnerability is remote exploitable but it's better to have covered it.


To be Honest, I threw the compromised thing as a possibility, but I'm very doubtful.
For an admin that's not a bad quality.
When in doubt I think checking stuff for starters is never bad.


It might just be a software conflict somewhere that is causing the system to reboot.
I'd also like to add that it just started happening the past couple of days.

Hmm. Check back your upgrade logs, make running apps log verbose, add a line like "*.* /var/log/catchall" to /etc/syslog.conf, touch the logfile and kill -HUP syslog is what I can suggest. Check the stuff I wrote about what files to check, even if it's not about compromises it still may gice you idea's.
If you're sure the're nothing logged before the syslog/klogd shutdown messages, maybe you could run atsar or sar and make sure it's not something with load, I/O, max processes, memory etc etc.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Solaris Installation - System Keeps Rebooting qs_tahmeed Solaris / OpenSolaris 3 07-18-2005 12:34 AM
problems after rebooting the system!! sharmashikha Linux From Scratch 2 07-09-2005 02:29 AM
System automatically rebooting! defa0009 Linux - General 13 03-23-2004 11:01 AM
System keeps rebooting after Kernel Loads codeviking Linux - Software 4 10-19-2003 09:02 PM
Self Rebooting System hubergeek Linux - General 2 06-11-2003 08:36 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 01:58 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration