Shell script to check if a port is in netstat then act on it

kayasaman · 01-05-2012, 09:38 PM

Hi,

I am trying to write a shell script which will take in the output of netstat and look for a specific port then determine that the correlating service is up and running or not. Then tie it into a cron job to automate.

What I have so far is this:

Code:

#!/bin/sh
ntstat=`netstat -ap tcp | grep 8180 | sed -n '1p'`
port=".8180*"

#echo $ntstat
#echo $port

if [ $ntstat =~ $port ]; then
   echo "Output of Netstat command $ntstat port number $port";
else
   wait 60; /usr/local/etc/rc.d/tomcat6 restart;
fi

It doesn't work and is far from working, hence I desperately need to improve my shell scripting capability but for now here's what I want the script to do in step order:

1. run netstat and output the desired port on one line - 8180 (Tomcat6)
2. store the information into a variable
3. compare the result of netstat with the port number 8180
4. conditionally if it's online do nothing
5. if it's not then wait 60 seconds and then restart tomcat

I'm having to build in some intelligence to my system as for whatever reason - probably lack of memory tomcat does segfault or run out of Java Heap Space - which I rectified.

Anyway it's a home server and I'm not in a position to get a memory hike as the machine's system board is maxed out and it's running on Mini-ITX hardware meaning only 4GB max for Core2Quad based system boards.

Thanks for any help!

grail · 01-05-2012, 11:56 PM

Have you tested your netstat command on the command line?

You have not said what errors, if any, that you are receiving?

Lastly, the posix shell does not support regular expression matching, =~, but bash does although you would also need to use double square brackets.

kayasaman · 01-06-2012, 12:09 AM

Quote:

Have you tested your netstat command on the command line?

You have not said what errors, if any, that you are receiving?

Lastly, the posix shell does not support regular expression matching, =~, but bash does although you would also need to use double square brackets.

When I run the script I directly get this:

Code:

[: tcp4: unexpected operator
wait: No such job: 60
Stopping tomcat60.
Waiting (max 10 secs) for PIDS: 83275, 83275, 83275, 83275.
Starting tomcat60.

which isn't right at all since Tomcat is already running!

The netstat command is fine:

Code:

# netstat -ap tcp | grep 8180 | sed -n '1p'
tcp4       0      0 wiki.8180              *.*                    LISTEN

The IF statement is well... iffy!

I will try your suggestion and report back.

kayasaman · 01-06-2012, 12:14 AM

Ok well switched my shell environment to Bash!

This output now:

Code:

Output of Netstat command tcp4       0      0 wiki.8180              proxy.45487            TIME_WAIT port number .8180*

The new code is as so:

Code:

#!/usr/local/bin/bash
ntstat=`netstat -ap tcp | grep 8180 | sed -n '1p'`
port=".8180*"

#echo $ntstat
#echo $port

if [[ $ntstat =~ $port ]]; then
   echo "Output of Netstat command $ntstat port number $port";
else 
   wait 60; /usr/local/etc/rc.d/tomcat6 restart;
fi

I'm running on FreeBSD hence the reason why Bash is under /usr/local as has been built from ports.

The only thing now is, is there a way to add white space after this:

port=".8180*"

??

Just so that the port that tomcat is running on gets detected and not an external port connecting to the system.

grail · 01-06-2012, 03:54 AM

Well I do not have tomcat running but your netstat seems to behave differently to mine

Here is an example:

Code:

$ netstat -ap tcp | grep 5656 | sed '1p'
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
unix  3      [ ]         STREAM     CONNECTED     5656     -                   
unix  3      [ ]         STREAM     CONNECTED     5656     -

The first 2 lines are actually printed to standard error so I just wanted to check with you

This seemed to solve it for me:

Code:

$ netstat -ap tcp 2>/dev/null | sed -n '/5656/p'
unix  3      [ ]         STREAM     CONNECTED     5656     -

Anyhoo, as far as the port idea goes, I am not sure why you have any superfluous characters in the test?
Why not simply set port variable to 8180?

Cedrik · 01-06-2012, 05:09 AM

Why test if output line contains 8180 in if condition, this number was already matched with grep

I would store the return value ($?) in a variable after the netstat command line and test if this variable = 0

kayasaman · 01-06-2012, 07:08 AM

I adapted the script a little:

Code:

#!/usr/local/bin/bash
ntstat=`netstat -ap tcp | grep 8180 | sed -n '1p'`
port=".8180*"

#echo $ntstat
#echo $port

if [[ $ntstat =~ $port ]]; then
   echo "Output of Netstat command $ntstat port number $port" > /root/java_restart/java_restart.log;
else 
   wait 60; /usr/local/etc/rc.d/tomcat6 restart;
fi

I chose .8180* as I wasn't quite sure if using 8180 would also print out other ports not used by tomcat as this is the general output of my netstat:

Code:

wiki# netstat -ap tcp
netstat: kvm not available: /dev/mem: No such file or directory
Active Internet connections (including servers)
Proto Recv-Q Send-Q  Local Address          Foreign Address       (state)
tcp4       0      0 wiki.8180              proxy.55891            ESTABLISHED
tcp4       0      0 wiki.postgresql        wiki.12899             ESTABLISHED
tcp4       0      0 wiki.12899             wiki.postgresql        ESTABLISHED
tcp4       0      0 wiki.8005              *.*                    LISTEN
tcp4       0      0 wiki.8009              *.*                    LISTEN
tcp4       0      0 wiki.8180              *.*                    LISTEN
tcp4       0      0 wiki.postgresql        *.*                    LISTEN
tcp4       0      0 wiki.smtp              *.*                    LISTEN

as I don't want to get the Foreign Address portion matched too.

I mean I wanted do the script the forward mentality way to test if the port was not on first then restart and if it was up then to just leave it. However my scripting skills are far from being perfect so I wasn't quite sure how to go about and it seemed easier to test the condition I used in the end.

Cedrik · 01-06-2012, 07:15 AM

Just a question

Code:

/usr/local/etc/rc.d/tomcat6 status

...does not work, to get tomcat6 status ?

kayasaman · 01-06-2012, 07:17 AM

It seems to:

Code:

wiki# /usr/local/etc/rc.d/tomcat6 status
tomcat60 is running as pid 20557.

But in all truth I didn't know about the command...

Cedrik · 01-06-2012, 07:21 AM

Sorry I just realized that if you have tomcat6 making segfault, surelly the pid file won't be updated and so the status command may show wrong info :/

kayasaman · 01-06-2012, 07:30 AM

Yeah actually it's strange that Tomcat6 is quite unstable on FreeBSD - am running 8.2 x64 edition.

Which is basically why I need to take extra percautions!

At first I was running out of Java Heap Space which I rectified and after telling the system to use 3GB it finally worked. Having said that I tested the same webapp (Xwiki) in Nexenta Core 3 with GlassFishV3 on a VM in my notebook with only 2GB memory allocated to it.

I ran for weeks and was fine - ok really slow but still.

I also tested on my Fedora 11 PPC based machine with only 384MB RAM which worked fine for a while but eventually swap space started getting colonized as Java needs a lot of memory to run on. But at least that instance even lasted for a week. On the most powerful system I have however (core2quad 4GB RAM), I keep getting problems. I think this should take care of most things though.

I can't simply reformat as the system is a sort'a home mainframe doing everything in one box and the reason behind FreeBSD in the first place was amount of apps readily available in repos plus ZFS file system. Not to mention really low memory footprint over Linux distro's I know plus the 4GB min that Solaris and it's variants need. + I like FreeBSD too :-) it's cool!

unSpawn · 01-06-2012, 07:41 AM

What I would do instead is query the process that's started from /usr/local/etc/rc.d/tomcat6 for its process id and check that process for ports it's listening on (or use say 'fuser -n tcp 8180' to isolate). In the end though a kludge that isn't in any way reliable as the port may be bound to but it won't return anything if the process is stuck and the service may not be accessible from remote hosts (OK, if that's required). An easy to install application like Monit has the advantage of 0) being faster than interpreted code, 1) has ready-made checks built in to test using process Id, ICMP, HTTP(S), content strings, content hashes using an easy to understand syntax and 2) can restart processes and alert you if anything occurs...

Cedrik · 01-06-2012, 07:43 AM

Waow, core2quad 4GB RAM and have stabilities issues ? Java is evil :/
In some way, I think I would consider testing an Open Solaris install maybe this java app would run smoother in it ? (java is SUN product after all)
(edit)
Just see that OpenSolaris project is discontinued :/

kayasaman · 01-06-2012, 07:59 AM

Thanks UnSpawn for that but Monit unfortunately takes memory!

I had to disable: Munin, Zabbix, Cacti, and basically any other non-essential service to make room for Tomcat.

The few things I can't disable are Bind, NTP, ZFS (which is a memory hog) and Squid. This box is also used as file server over NFS. - also am running FreeBSD Jail'ed environment which is effectively similar to running Solaris Containers which also takes memory.

Cedrik, yeah!!

Here is top output from my system:

Code:

Mem: 2046M Active, 272M Inact, 1367M Wired, 47M Cache, 408M Buf, 96M Free
Swap: 2327M Total, 1870M Used, 456M Free, 80% Inuse

  PID JID USERNAME   THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
21242  22 www         50  44    0  3565M  1611M ucond   0  18:32  0.00% java
20557  14 www         43  76    0  3528M   392M ucond   0   1:21  0.00% java

nearly 2GB of swap used!!!

Yeah Solaris would have been cool but OpenSolaris is DEAD! - Thanks Oracle. Nexenta Core 3 would have been the way to go but I knew that Munin, Zabbix, and Cacti were readily and easily available in FreeBSD so I went down that route first. Only later I decided to run Xwiki in Tomcat when I started getting the errors.

If I manage to get employed this year I plan on building a dedicated box for my wiki anyway using Solaris 11 or Nexenta Core 3 (undecided currently) with GlassFishV3 and install about 8GB memory. That should be fine then as I can run multiple 'Solaris Zones' then have one DB instance for postgresql and 2 for my seperate instances of tomcat.

But as stated it all depends on cash flow. :-(

unSpawn · 01-06-2012, 01:57 PM

Quote:

Originally Posted by kayasaman

Thanks UnSpawn for that but Monit unfortunately takes memory!

Heh, not using any memory wasn't a posted requirement ;-p

Quote:

Originally Posted by kayasaman

I had to disable (..) basically any other non-essential service to make room for Tomcat (..) Bind, NTP, ZFS (..) Squid. (..) file server over NFS (..) FreeBSD Jail'ed environment (..) nearly 2GB of swap used

Once processes start swapping constantly things spin out of control. Java eats RAM like it's 1999, Squid needs RAM for caching (expensive I/O), too many services on that machine IMHO. Unless you're infatuated with Java or require specific Xwiki functionality you can't get otherwise why not try Wiki SW that doesn't hog memory?..

Quote:

Originally Posted by kayasaman

Solaris would have been cool but OpenSolaris is DEAD

Ever heard of OpenIndiana?