Too many file open - How can we define Max limit of file descriptor

janeshb · 07-07-2009, 02:27 AM

We are facing problem of to many file open error because of that application become slow and in tomcat catalina log we get following error frequently

Jul 6, 2009 12:27:57 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:6 1)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310)
at java.lang.Thread.run(Thread.java:619)

What is the ideal no of file limit,file descriptor limit for 300 user of tomcat application server and also oracle database server?

And how to set and where??

acid_kewpie · 07-07-2009, 02:41 AM

at a system wide level, define it in /etc/sysctl.conf (then run sysctl -p /etc/sysctl.conf). This is normally 65535 or something like that. per user, run "ulimit -n" to see what each user has, default usually something like 1024, so something like "ulimit -n 8192" might be in order - which is permanently set in /etc/security/limits.conf on most systems.

janeshb · 07-07-2009, 02:56 AM

Thank you for quick reply.

We already have set in /etc/sysctl.conf file-max = 700000
and also set /etc/security/limits.conf soft = 700000 hard = 700000

but still error exist..

acid_kewpie · 07-07-2009, 04:18 AM

as the user running the process, e.g. tomcat4, run "ulimit -a"

janeshb · 07-07-2009, 08:52 AM

tomcat running as root only..

acid_kewpie · 07-07-2009, 11:06 AM

can you show us the outpu then? ulimit -n, cat /proc/sys/fs/file-max and also the ps -ef output showing the tomcat processes

janeshb · 07-08-2009, 01:30 AM

ulimit -n
1000000

cat /proc/sys/fs/file-nr
1530 0 700000

ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Jul07 ? 00:00:01 init [5]
root 2 1 0 Jul07 ? 00:00:00 [migration/0]
root 3 1 0 Jul07 ? 00:00:00 [ksoftirqd/0]
root 4 1 0 Jul07 ? 00:00:00 [watchdog/0]
root 5 1 0 Jul07 ? 00:00:00 [migration/1]
root 6 1 0 Jul07 ? 00:00:00 [ksoftirqd/1]
root 7 1 0 Jul07 ? 00:00:00 [watchdog/1]
root 8 1 0 Jul07 ? 00:00:00 [migration/2]
root 9 1 0 Jul07 ? 00:00:00 [ksoftirqd/2]
root 10 1 0 Jul07 ? 00:00:00 [watchdog/2]
root 11 1 0 Jul07 ? 00:00:00 [migration/3]
root 12 1 0 Jul07 ? 00:00:00 [ksoftirqd/3]
root 13 1 0 Jul07 ? 00:00:00 [watchdog/3]
root 14 1 0 Jul07 ? 00:00:00 [events/0]
root 15 1 0 Jul07 ? 00:00:00 [events/1]
root 16 1 0 Jul07 ? 00:00:00 [events/2]
root 17 1 0 Jul07 ? 00:00:00 [events/3]
root 18 1 0 Jul07 ? 00:00:00 [khelper]
root 57 1 0 Jul07 ? 00:00:00 [kthread]
root 64 57 0 Jul07 ? 00:00:00 [kblockd/0]
root 65 57 0 Jul07 ? 00:00:00 [kblockd/1]
root 66 57 0 Jul07 ? 00:00:00 [kblockd/2]
root 67 57 0 Jul07 ? 00:00:00 [kblockd/3]
root 68 57 0 Jul07 ? 00:00:00 [kacpid]
root 177 57 0 Jul07 ? 00:00:00 [cqueue/0]
root 178 57 0 Jul07 ? 00:00:00 [cqueue/1]
root 179 57 0 Jul07 ? 00:00:00 [cqueue/2]
root 180 57 0 Jul07 ? 00:00:00 [cqueue/3]
root 183 57 0 Jul07 ? 00:00:00 [khubd]
root 185 57 0 Jul07 ? 00:00:00 [kseriod]
root 272 57 0 Jul07 ? 00:00:00 [pdflush]
root 273 57 0 Jul07 ? 00:00:00 [pdflush]
root 274 57 0 Jul07 ? 00:00:00 [kswapd0]
root 275 57 0 Jul07 ? 00:00:00 [aio/0]
root 276 57 0 Jul07 ? 00:00:00 [aio/1]
root 277 57 0 Jul07 ? 00:00:00 [aio/2]
root 278 57 0 Jul07 ? 00:00:00 [aio/3]
root 437 57 0 Jul07 ? 00:00:00 [kpsmoused]
root 499 57 0 Jul07 ? 00:00:00 [scsi_eh_0]
root 505 57 0 Jul07 ? 00:00:00 [ata/0]
root 506 57 0 Jul07 ? 00:00:00 [ata/1]
root 507 57 0 Jul07 ? 00:00:00 [ata/2]
root 508 57 0 Jul07 ? 00:00:00 [ata/3]
root 509 57 0 Jul07 ? 00:00:00 [ata_aux]
root 515 57 0 Jul07 ? 00:00:00 [scsi_eh_1]
root 516 57 0 Jul07 ? 00:00:00 [scsi_eh_2]
root 542 57 0 Jul07 ? 00:00:00 [ksnapd]
root 545 57 0 Jul07 ? 00:00:01 [kjournald]
root 573 57 0 Jul07 ? 00:00:00 [kauditd]
root 607 1 0 Jul07 ? 00:00:00 /sbin/udevd -d
root 1802 57 0 Jul07 ? 00:00:00 [kmpathd/0]
root 1803 57 0 Jul07 ? 00:00:00 [kmpathd/1]
root 1804 57 0 Jul07 ? 00:00:00 [kmpathd/2]
root 1806 57 0 Jul07 ? 00:00:00 [kmpathd/3]
root 1831 57 0 Jul07 ? 00:00:00 [kjournald]
root 2159 1 0 Jul07 ? 00:00:00 syslogd -m 0
root 2162 1 0 Jul07 ? 00:00:00 klogd -x
root 2174 1 0 Jul07 ? 00:00:00 irqbalance
dbus 2186 1 0 Jul07 ? 00:00:00 dbus-daemon --system
root 2224 1 0 Jul07 ? 00:00:00 automount
root 2246 1 0 Jul07 ? 00:00:00 /usr/sbin/sshd
root 2258 1 0 Jul07 ? 00:00:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
root 2269 1 0 Jul07 ? 00:00:00 crond
xfs 2300 1 0 Jul07 ? 00:00:00 xfs -droppriv -daemon
root 2311 1 0 Jul07 ? 00:00:00 smbd -D
root 2314 1 0 Jul07 ? 00:00:03 nmbd -D
root 2322 2311 0 Jul07 ? 00:00:00 smbd -D
zabbix 2336 1 0 Jul07 ? 00:00:00 /usr/local/sbin/zabbix_agentd
zabbix 2339 2336 0 Jul07 ? 00:00:00 /usr/local/sbin/zabbix_agentd
zabbix 2340 2336 0 Jul07 ? 00:00:00 /usr/local/sbin/zabbix_agentd
zabbix 2341 2336 0 Jul07 ? 00:00:00 /usr/local/sbin/zabbix_agentd
zabbix 2342 2336 0 Jul07 ? 00:00:00 /usr/local/sbin/zabbix_agentd
zabbix 2343 2336 0 Jul07 ? 00:00:00 /usr/local/sbin/zabbix_agentd
root 2614 1 0 Jul07 ? 00:00:00 [cifsoplockd]
root 2615 1 0 Jul07 ? 00:00:00 [cifsdnotifyd]
root 2624 1 0 Jul07 ? 00:00:00 [cifsd]
root 2636 1 0 Jul07 ? 00:00:00 /usr/sbin/smartd -q never
root 2641 1 0 Jul07 tty1 00:00:00 /sbin/mingetty tty1
root 2642 1 0 Jul07 tty2 00:00:00 /sbin/mingetty tty2
root 2643 1 0 Jul07 tty3 00:00:00 /sbin/mingetty tty3
root 2644 1 0 Jul07 tty4 00:00:00 /sbin/mingetty tty4
root 2645 1 0 Jul07 tty5 00:00:00 /sbin/mingetty tty5
root 2646 1 0 Jul07 tty6 00:00:00 /sbin/mingetty tty6
root 2647 1 0 Jul07 ? 00:00:00 /usr/sbin/gdm-binary -nodaemon
root 2742 2647 0 Jul07 ? 00:00:00 /usr/sbin/gdm-binary -nodaemon
root 2745 2742 0 Jul07 tty7 00:00:00 /usr/bin/Xorg :0 -br -audit 0 -auth /var/gdm/:0.Xauth -nolisten tcp vt7
gdm 2764 2742 0 Jul07 ? 00:00:00 /usr/libexec/gdmlogin
root 5937 1 4 Jul07 ? 00:44:16 /opt/jdk1.6.0_07/bin/java -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/opt/apache-tomcat-6.0.16/conf/logging.properties -Xms512m -Xmx2048m -XX:PermSize=512m -XX:MaxPermSize=2048m -Djava.endorsed.dirs=/opt/apache-tomcat-6.0.16/endorsed -classpath :/opt/apache-tomcat-6.0.16/bin/bootstrap.jar -Dcatalina.base=/opt/apache-tomcat-6.0.16 -Dcatalina.home=/opt/apache-tomcat-6.0.16 -Djava.io.tmpdir=/opt/apache-tomcat-6.0.16/temp org.apache.catalina.startup.Bootstrap start
root 9027 2269 0 10:00 ? 00:00:00 crond
root 9028 9027 0 10:00 ? 00:00:00 /bin/sh -c sh /cron_script/dataupload.sh > /var/log/dataupload.log
root 9029 9028 0 10:00 ? 00:00:00 sh /cron_script/dataupload.sh
root 9031 9029 0 10:00 ? 00:00:11 java -classpath /opt/apache-tomcat-6.0.16/lib/commons-codec-1.3.jar:/opt/apache-tomcat-6.0.16/lib/commons-httpclient-3.0.1.jar:/opt/apache-tomcat-6.0.16/lib/commons-logging-1.1.jar:/opt/apache-tomcat-6.0.16/lib/commons-logging-api-1.1.jar:/opt/apache-tomcat-6.0.16/lib/commons-logging-adapters-1.1.jar:/opt/apache-tomcat-6.0.16/lib/commons-httpclient-3.0.1:/opt/apache-tomcat-6.0.16/lib/ojdbc14.jar:/opt/apache-tomcat-6.0.16/lib/finutils.jar:/opt/apache-tomcat-6.0.16/webapps/DataUploadV3.0/src:/opt/apache-tomcat-6.0.16/old_lib/mysql-connector-java-3.1.6-bin.jar:/opt/apache-tomcat-6.0.16/webapps/DataUploadV3.0/src Main TRANSACTION
root 9421 5937 0 10:41 ? 00:00:00 /opt/jdk1.6.0_07/bin/java -classpath /opt/apache-tomcat-6.0.16/webapps/mfund/WEB-INF/classes:/opt/apache-tomcat-6.0.16/lib/ojdbc14.jar:/opt/apache-tomcat-6.0.16/lib/finutils.jar:/opt/apache-tomcat-6.0.16/lib/servlet-api.jar NJUtility.DBUtility.brokerage_auto_process_main
root 9452 2246 0 10:48 ? 00:00:00 sshd: janesh [priv]
janesh 9455 9452 0 10:48 ? 00:00:00 sshd: janesh@pts/0
janesh 9456 9455 0 10:48 pts/0 00:00:00 -bash
root 9479 9456 0 10:48 pts/0 00:00:00 su
root 9483 9479 0 10:48 pts/0 00:00:00 bash
root 9559 2311 0 10:57 ? 00:00:00 smbd -D
nobody 9567 2311 0 10:59 ? 00:00:00 smbd -D
root 9569 9483 0 10:59 pts/0 00:00:00 ps -ef

suhas! · 07-08-2009, 02:35 AM

how many file descriptors are open by tomcat process when error is encountered?
# lsof -p <pid of tomcat> | wc -l

janeshb · 07-09-2009, 05:50 AM

# lsof -p <pid of tomcat> | wc -l

1029

janeshb · 07-14-2009, 01:46 AM

# lsof -p <pid of tomcat> | wc -l

5001

janeshb · 09-29-2009, 09:23 AM

In tomcat application server all application and user run through one process only. So when process access 1024 this too many open file error will come. This error has not solved even if we change in file-max or in security.conf file soft/hard limit.

So how can we change this 1024 per process limit.

jeffbarron · 09-29-2009, 11:41 AM

Quote:

Originally Posted by janeshb

In tomcat application server all application and user run through one process only. So when process access 1024 this too many open file error will come. This error has not solved even if we change in file-max or in security.conf file soft/hard limit.

So how can we change this 1024 per process limit.

I was wondering if maybe its a concurrency bug in the app you're running. I skimmed the post but the first post I saw "threads" and "a too many" error which first makes me think of threads gone wild. If your question was about Tomcat and not a specific app then disregard my comments.

Jeff