LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 11-06-2011, 12:24 PM   #1
TheOnlyQ
Member
 
Registered: Apr 2011
Posts: 33

Rep: Reputation: 0
Scripting a network restart when net goes down.


Alright, so as a first-action solution I've come up with an idea.

Basically every now and then we get very quick short-lasting incoming ddos attacks and it seems the NIC gives up but the server is still online. So to counteract this I thought it would be good to have a script which checks if it can ping google.com or any other sustainable host.. this script can run via cron every 5minutes.

When it can't contact google.com and has already tried 2 times, it restarts the network with 'service network restart' and then removes the log of its past two attempts.

I'm just wondering, how would I allow the script to check if the ping is succeeding? I'm not great at bash scripting so I would appreciate any help.

Thanks!
 
Old 11-06-2011, 01:16 PM   #2
spiky0011
Senior Member
 
Registered: Jan 2011
Location: PLANET-SPIKE
Distribution: /LFS/Centos6.3/Debian
Posts: 2,127
Blog Entries: 1

Rep: Reputation: 316Reputation: 316Reputation: 316Reputation: 316
How about
Code:
ping -i120 google.com
 
Old 11-06-2011, 01:17 PM   #3
ButterflyMelissa
Senior Member
 
Registered: Nov 2007
Location: Somewhere on my hard drive...
Distribution: Manjaro
Posts: 2,686
Blog Entries: 23

Rep: Reputation: 398Reputation: 398Reputation: 398Reputation: 398
Hmm,

Maybe this could help, dunnow...

Quote:
#!/bin/bash

ping -c 2 www.google.com > /dev/null
if [ $? -ne 0 ]; then
echo "restarting network"
/etc/init.d/networking restart
fi
Save it somewhere and put the command in the proper cron folder, but...you know this...

needs root rights, though...

Of course, the "bacon" of the thing is the line where the restart happens, depends on your distro, on mine (Arch Linux) this could be:

Quote:
/etc/rc.d/network restart
therefore, check the manual...

Thor

Last edited by ButterflyMelissa; 11-06-2011 at 01:26 PM.
 
Old 11-06-2011, 01:24 PM   #4
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,258

Rep: Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947
I wrote a script to do exactly this on one of our embedded systems a while back, works great.

The basic gist is to ping a known host (I usually use the router's IP since you don't want to go around restarting the network if it's just an Internet outage) and check the exit code like Thor showed. If the ping succeeds you exit, if it fails you wait a minute and try again. If it succeeds you exit, if it fails you wait a minute and try again. If it fails the third time, you write the event to a log and issue a network restart command.

Last edited by suicidaleggroll; 11-06-2011 at 01:26 PM.
 
Old 11-06-2011, 02:52 PM   #5
TheOnlyQ
Member
 
Registered: Apr 2011
Posts: 33

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by suicidaleggroll View Post
I wrote a script to do exactly this on one of our embedded systems a while back, works great.

The basic gist is to ping a known host (I usually use the router's IP since you don't want to go around restarting the network if it's just an Internet outage) and check the exit code like Thor showed. If the ping succeeds you exit, if it fails you wait a minute and try again. If it succeeds you exit, if it fails you wait a minute and try again. If it fails the third time, you write the event to a log and issue a network restart command.
For learning purposes would you mind sharing your script?

I'm definitely not any good in bash so I'm basing this from piecing bits together, and I don't think that method works all to well so I would like to learn from something whole if you would be kind enough.
 
Old 11-06-2011, 04:10 PM   #6
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,258

Rep: Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947
Keep in mind that this was developed for an embedded debian system with a lot of restrictions, which is why it's written the way it is (no cron, etc). It should run on a regular system as well, but in that case I would probably make a few changes.

It's invoked in rc.local using nohup so it's always running in the background. With cron this wouldn't be necessary.

Code:
#!/bin/bash

last=0
last2=0

while [[ 1 ]]; do
   source ethconfig

   ping -c 1 $ETH_GW > /dev/null
   status=$?
   if [[ $status == 1 ]]; then
      echo "ping failed" >> eth_watchdog.log
      if [[ $last == 1 && $last2 == 1 ]]; then
         echo "calling /etc/init.d/networking restart" >> eth_watchdog.log
         echo `date` >> eth_watchdog.log
         /etc/init.d/networking restart
         status=0
      fi
   fi
   last2=$last
   last=$status

   sleep $ETHWD_SLEEP
done
ethconfig looks something like:
Code:
export ETHWD_SLEEP=60
export ETH_GW=192.168.1.1
 
Old 11-07-2011, 12:48 PM   #7
TheOnlyQ
Member
 
Registered: Apr 2011
Posts: 33

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by suicidaleggroll View Post
Keep in mind that this was developed for an embedded debian system with a lot of restrictions, which is why it's written the way it is (no cron, etc). It should run on a regular system as well, but in that case I would probably make a few changes.

It's invoked in rc.local using nohup so it's always running in the background. With cron this wouldn't be necessary.

Code:
#!/bin/bash

last=0
last2=0

while [[ 1 ]]; do
   source ethconfig

   ping -c 1 $ETH_GW > /dev/null
   status=$?
   if [[ $status == 1 ]]; then
      echo "ping failed" >> eth_watchdog.log
      if [[ $last == 1 && $last2 == 1 ]]; then
         echo "calling /etc/init.d/networking restart" >> eth_watchdog.log
         echo `date` >> eth_watchdog.log
         /etc/init.d/networking restart
         status=0
      fi
   fi
   last2=$last
   last=$status

   sleep $ETHWD_SLEEP
done
ethconfig looks something like:
Code:
export ETHWD_SLEEP=60
export ETH_GW=192.168.1.1
Yeah, having a hard time converting this to CentOS. Any pro's got any tips?
 
Old 11-07-2011, 01:47 PM   #8
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,258

Rep: Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947
What kind of problems are you having? I just ran it without any issues on Redhat Enterprise 4 and Fedora 15.

Last edited by suicidaleggroll; 11-07-2011 at 01:50 PM.
 
Old 11-07-2011, 03:02 PM   #9
TheOnlyQ
Member
 
Registered: Apr 2011
Posts: 33

Original Poster
Rep: Reputation: 0
sleep: missing operand
Try `sleep --help' for more information.
./test.sh: line 7: ethconfig: No such file or directory
Usage: ping [-LRUbdfnqrvVaA] [-c count] [-i interval] [-w deadline]
[-p pattern] [-s packetsize] [-t ttl] [-I interface or address]
[-M mtu discovery hint] [-S sndbuf]
[ -T timestamp option ] [ -Q tos ] [hop1 ...] destination


I've tried changing ethconfig to ifconfig but commands are different.

Ping, not sure about that.

Sleep, the same, not sure. Seems your script would work but I just need to know how to convert several parts to centos.
 
Old 11-07-2011, 03:50 PM   #10
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,258

Rep: Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947
ethconfig is an ascii file with two lines as shown, it tells the script which IP to ping and how long to wait between repeat attempts. The script is failing on line 7 because you didn't create the ethconfig file. Since you didn't create the ethconfig file, that means the $ETH_GW and $ETHWD_SLEEP variables are empty, which is why the calls to ping and sleep are failing. Create the ethconfig file in your pwd (or create it elsewhere and hard-code the location in the script) and all of those problems will go away.

Alternatively, you can just swap the values in ethconfig into the script directly and remove the "source ethconfig" line. I didn't do that in my version because this code is always running, 24/7, so if I ever wanted to change the IP or the delay time, I would have to kill the script, change the values, and then re-start it. Separating those values into their own file allows me to change them without having to kill and restart the script (which is why ethconfig is being sourced every iteration of the loop). If you convert this script into a cron version (aka: not an infinite loop), then those problems become moot.

Last edited by suicidaleggroll; 11-07-2011 at 03:53 PM.
 
Old 11-08-2011, 04:44 AM   #11
TheOnlyQ
Member
 
Registered: Apr 2011
Posts: 33

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by suicidaleggroll View Post
ethconfig is an ascii file with two lines as shown, it tells the script which IP to ping and how long to wait between repeat attempts. The script is failing on line 7 because you didn't create the ethconfig file. Since you didn't create the ethconfig file, that means the $ETH_GW and $ETHWD_SLEEP variables are empty, which is why the calls to ping and sleep are failing. Create the ethconfig file in your pwd (or create it elsewhere and hard-code the location in the script) and all of those problems will go away.

Alternatively, you can just swap the values in ethconfig into the script directly and remove the "source ethconfig" line. I didn't do that in my version because this code is always running, 24/7, so if I ever wanted to change the IP or the delay time, I would have to kill the script, change the values, and then re-start it. Separating those values into their own file allows me to change them without having to kill and restart the script (which is why ethconfig is being sourced every iteration of the loop). If you convert this script into a cron version (aka: not an infinite loop), then those problems become moot.
Alright, it is now working, from what I can tell it is registering that the ping is failing. watchdog log >


ping failed
ping failed
ping failed
ping failed


However it doesn't seem to be restarting the network, including it isn't logging that it is even attempting it, or echo'ing it.

Any advice? Once again appreciate it.
 
Old 11-08-2011, 04:52 AM   #12
d3vrandom
Member
 
Registered: Jun 2006
Location: Karachi, Pakistan
Distribution: OpenSUSE, CentOS, Debian
Posts: 59

Rep: Reputation: 9
I suggest using monit for this sort of thing. That is what it's designed to do.
 
Old 11-08-2011, 04:54 AM   #13
TheOnlyQ
Member
 
Registered: Apr 2011
Posts: 33

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by d3vrandom View Post
I suggest using monit for this sort of thing. That is what it's designed to do.
Can you provide any more info? The script above I have working and it seems great but it isn't restarting the network, trying to work out why.
 
Old 11-08-2011, 09:02 AM   #14
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,258

Rep: Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947
Quote:
Originally Posted by TheOnlyQ View Post
Alright, it is now working, from what I can tell it is registering that the ping is failing. watchdog log >


ping failed
ping failed
ping failed
ping failed


However it doesn't seem to be restarting the network, including it isn't logging that it is even attempting it, or echo'ing it.

Any advice? Once again appreciate it.
If you look closely at the script, you'll see it only restarts the network if the ping has failed 3 times in a row. If it just fails once or twice, it will write "ping failed" to the output, but as long as the ping succeeds the third time it won't restart the network. Throw a date in the log file next to the "ping failed" echo and you'll probably see that your "ping failed" prints aren't in a row, they're random and separated by some time, which is why it's not taking action.
 
Old 11-08-2011, 01:24 PM   #15
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire
Distribution: Debian Wheezy/Jessie/Sid, Linux Mint DE
Posts: 4,493

Rep: Reputation: 635Reputation: 635Reputation: 635Reputation: 635Reputation: 635Reputation: 635
Don't do like I did!

Once I put a script on my client of my systems which rebooted the machine when the network was down. I can't remember why I had the machine reboot instead of simply restarting the network. Maybe it had something to do with module loading. I must have installed it back in 2005 or so. I renewed some system components but cloned the installation when replacing the hard disk.

Anyway, years went by and (as usual with Linux) I never had a problem, and (as usual with Linux) things were working so well I had totally forgotten about installing that script. I even forgot that I ever wrote it.

What did this script do? Exactly 10 minutes after booting the client would ping our main server. That was the WinNT (at that time) PDC, and if that one was down, there would be something very wrong anyway, so that looked a reliable source. The PDC never went down and there were no problems at all.

Until 2 years ago I replaced that WinNT box with an all new Linux server. With a different IP address of course because the two servers would need to run side by side for a while. The the WinNT server was taken off-line.

Nothing happened, because my client only checked 10 minutes after booting. Not every 10 minutes. So weeks after taking the WinNT server off-line I rebooted the client. Which faithfully rebooted after 10 minutes.

Not only could I remember ever installing the script, there was also no association with the server going off-line. There were weeks in between.

So what does one do with a desktop computer which reboots after 10 minutes? Right, suspect the cooling, clean the fan, replace it, check the power supply, replace it, memory, hard disk.... Long story short, I can't remember anymore how I discovered the script again, but after replacing most of the hardware I finally did. IIRC I logged an entry in the /var/log/messages, but at first I didn't understand why a network failure would force a reboot.

Anyway, don't do it my way. It was stupid.

jlinkels

Last edited by jlinkels; 11-08-2011 at 01:26 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
800Mb debian anyone / how to restart net connection without restarting comp? redlinuxxx Debian 9 09-21-2010 08:56 PM
net drops while others in the same network download something or browse the net !! wanas Linux - Networking 9 10-09-2009 07:53 PM
scripting help: how to restart an application that fails to start sometimes? murray_linux Linux - Software 16 06-27-2007 04:13 PM
restart needed for connecting to net. j_hawk Linux - Networking 6 02-19-2007 12:48 PM
How do I set the regulatory daemon to restart when I restart the network service? zahadumy Linux - Networking 0 11-05-2006 12:24 PM


All times are GMT -5. The time now is 06:34 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration