LinuxQuestions.org
Visit Jeremy's Blog.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 08-26-2008, 04:03 PM   #1
twlilinux
Member
 
Registered: Jan 2006
Distribution: debian
Posts: 63

Rep: Reputation: 15
troubleshoot: my server goes down. Comes back up after reboot


Hi all, I have a remote, unmanaged dedicated Debian server. Last Tuesday, my server went down so I emailed the support team to reboot my server and it fixed the problem. Today, it happened again. And like last time, a reboot fixed the problem. I would really appreciate it if someone helps me to troubleshoot:

-------Netstat -lpn output--------:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 2135/mysqld
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 2394/dnscache
tcp6 0 0 :::ssh :::* LISTEN 2290/sshd
tcp6 0 0 :::80 :::* LISTEN 2337/apache2
tcp6 0 0 :::443 :::* LISTEN 2337/apache2
tcp6 0 576 ::ffff:ip:ssh ::ffff:myip:ssh ESTABLISHED2404/sshd: user [p
udp 0 0 127.0.0.1:53 0.0.0.0:* 2394/dnscache
udp 0 0 ip:53 0.0.0.0:* 2392/tinydns

-------My firewall output----------:

Accepting port ssh...
Accepting port 53...
Accepting port 80...
Accepting port 443...
blocking everything not listed here...
activating DoS prevention...
blocking XMAS-tree and NULL packets

-------My firewall--------:

#!/bin/sh
#
# A simple firewall initialization script
#
WHITELIST=/usr/local/etc/whitelist.txt
BLACKLIST=/usr/local/etc/blacklist.txt
ALLOWED="ssh 53 80 443"

#
# Drop all existing filter rules
#
iptables -F

#
# First, run through $WHITELIST, accepting all traffic from hosts and networks
# contained therein.
#
for x in `grep -v ^# $WHITELIST | awk '{print $1}'`; do
echo "Permitting $x..."
iptables -A INPUT -t filter -s $x -j ACCEPT
done

#
# Now run through $BLACKLIST, dropping all traffic from the hosts and networks
# contained therein
#
for x in `grep -v ^# $BLACKLIST | awk '{print $1}'`; do
echo "Blocking $x..."
iptables -A INPUT -t filter -s $x -j DROP
done

#
# Next, the permitted ports: What will we accept from hosts not appearing
# on the blacklist?
#
for port in $ALLOWED; do
echo "Accepting port $port..."
iptables -A INPUT -t filter -p tcp --dport $port -j ACCEPT
done

#
# Finally, unless it's mentioned above, and it's an inbound startup request,
# just drop it.
#
echo "blocking everything not listed here..."
iptables -A INPUT -t filter -p tcp --syn -j DROP

#
# a simple DoS prevention script
#
echo "activating DoS prevention..."
# Create syn-flood chain for detecting Denial of Service attacks
iptables -t nat -N syn-flood

# Limit 12 connections per second (burst to 24)
iptables -t nat -A syn-flood -m limit --limit 12/s --limit-burst 24 -j RETURN
iptables -t nat -A syn-flood -j DROP

# Check for DoS attack
iptables -t nat -A PREROUTING -i $EXT_IFACE -d $DEST_IP -p tcp --syn -j syn-flood

#
# a simple script for blocking XMAS-tree and NULL packets
#
echo "blocking XMAS-tree and NULL packets"
iptables -t nat -A PREROUTING -p tcp --tcp-flags ALL ALL -j DROP
iptables -t nat -A PREROUTING -p tcp --tcp-flags ALL NONE -j DROP
 
Old 08-26-2008, 04:39 PM   #2
trickykid
LQ Guru
 
Registered: Jan 2001
Posts: 24,149

Rep: Reputation: 269Reputation: 269Reputation: 269
Posting your firewall configs and netstat isn't going to tell us anything. We need server logs to determine the issue. Also, is this a real dedicated server or a virtual server?
 
Old 08-26-2008, 05:44 PM   #3
twlilinux
Member
 
Registered: Jan 2006
Distribution: debian
Posts: 63

Original Poster
Rep: Reputation: 15
Thanks for the reply- It's a dedicated server, I have root access. What logs would you recommend checking? and for what kind of information?
 
Old 08-26-2008, 06:07 PM   #4
Poetics
Senior Member
 
Registered: Jun 2003
Location: California
Distribution: Slackware
Posts: 1,181

Rep: Reputation: 49
You say the server itself went down. Did it go down or just it's networking? Did the people at your hosting company plug a console into it? If so, did they record any messages they saw?

What do the various logs in /var/log say around the time of the event?
 
Old 08-26-2008, 06:18 PM   #5
twlilinux
Member
 
Registered: Jan 2006
Distribution: debian
Posts: 63

Original Poster
Rep: Reputation: 15
hey, thanks for the tip. I checked the /var/log's syslog file and found the following entries around the time of the problem that looked suspicious.

I'm almost certain that the "server kernel" entries is the smoking gun. Anyone have a clue what it means?


--------syslog---------:
Aug 26 14:30:01 server /USR/SBIN/CRON[13708]: (www-data) CMD ([ -x /usr/lib/cgi-bin/awstats.pl -a -f /etc/awstats/awstats.conf -a -r /var/log/apache/access.log ] && /usr/lib/cgi-bin/awstats.pl -config=awstats -update >/dev/null)
Aug 26 14:36:38 server kernel: irq 169: nobody cared (try booting with the "irqpoll" option)
Aug 26 14:36:38 server kernel: [<c014030b>] __report_bad_irq+0x2b/0x69
Aug 26 14:36:38 server kernel: [<c01404f8>] note_interrupt+0x1af/0x1e7
Aug 26 14:36:38 server kernel: [<c013faff>] handle_IRQ_event+0x23/0x49
Aug 26 14:36:38 server kernel: [<c013fbd8>] __do_IRQ+0xb3/0xe8
Aug 26 14:36:38 server kernel: [<c01050e5>] do_IRQ+0x43/0x52
Aug 26 14:36:38 server kernel: [<c01036b6>] common_interrupt+0x1a/0x20
Aug 26 14:36:38 server kernel: [<c0101b91>] mwait_idle+0x25/0x38
Aug 26 14:36:38 server kernel: [<c0101b52>] cpu_idle+0x9f/0xb9
Aug 26 14:36:38 server kernel: [<c03196fd>] start_kernel+0x379/0x380
Aug 26 14:36:38 server kernel: handlers:
Aug 26 14:36:38 server kernel: [<f885947b>] (usb_hcd_irq+0x0/0x50 [usbcore])
Aug 26 14:36:38 server kernel: [<f885947b>] (usb_hcd_irq+0x0/0x50 [usbcore])
Aug 26 14:36:38 server kernel: [<f881cd12>] (rtl8169_interrupt+0x0/0x1cd [r8169])
Aug 26 14:36:38 server kernel: Disabling IRQ #169
Aug 26 14:39:01 server /USR/SBIN/CRON[13711]: (root) CMD ( [ -d /var/lib/php4 ] && find /var/lib/php4/ -type f -cmin +$(/usr/lib/php4/maxlifetime) -print0 | xargs -r -0 rm)
Aug 26 14:39:01 server /USR/SBIN/CRON[13713]: (root) CMD ( [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 | xargs -r -0 rm)
Aug 26 14:40:01 server /USR/SBIN/CRON[13732]: (www-data) CMD ([ -x /usr/lib/cgi-bin/awstats.pl -a -f /etc/awstats/awstats.conf -a -r /var/log/apache/access.log ] && /usr/lib/cgi-bin/awstats.pl -config=awstats -update >/dev/null)
Aug 26 14:50:01 server /USR/SBIN/CRON[13734]: (www-data) CMD ([ -x /usr/lib/cgi-bin/awstats.pl -a -f /etc/awstats/awstats.conf -a -r /var/log/apache/access.log ] && /usr/lib/cgi-bin/awstats.pl -config=awstats -update >/dev/null)
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
iptables firewall falls back to default after reboot lx3000 Linux - Security 8 03-31-2006 04:02 PM
monitor goes black, will not respond. have to reboot to get back in pjbii Slackware 1 03-13-2005 11:37 AM
Venting : Back From ReBoot-HELL.... Megamieuwsel General 7 10-02-2004 06:19 AM
How do I Reboot into Linux at night and back to WinXP in the morning? kayhan Linux - General 5 09-16-2004 04:45 PM
Server fails to reboot when power is back on edmunthali Red Hat 0 03-07-2004 01:35 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 03:03 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration