Too many open files in system

dreamchaser · 05-09-2015, 06:18 PM

Hi All,

Batch converting a large number of files from wav to mp3 with the following command eventually terminates with a "Too many open files in system" error:

"find /recordings/arrow82/monitor/2015/04 -name '*.wav' -execdir sh -c 'lame --quiet --preset phone '{}' '{}'.mp3' \;"

I increased the limits in /etc/security/limits.conf to:

root hard nofile 999999
root soft nofile 999999

...that helped for a while but there are now 200k+ files in the directory in which find is running.

Can the open files limit be increased further, or are there better ways of processing files in a directory tree?

rknichols · 05-09-2015, 07:18 PM

The problem is the find command trying to retain a handle on the working directory for each file that is used in an execdir. You will need to either run separate find commands for different subsets of your .wav files, or else use "-exec" instead of "-execdir".

metaschima · 05-09-2015, 07:22 PM

What you probably want is to convert the files a set number at a time.

One file at a time:

Code:

find /recordings/arrow82/monitor/2015/04 -type f -name '*.wav' | while read line
do 
    lame --quiet --preset phone "$line" "$(basename "$line" .wav).mp3"
done

You can run this a few at a time per directory.

I'll have to think of a clever way to run more than one at a time.

EDIT:

Code:

num_threads=4

find /recordings/arrow82/monitor/2015/04 -type f -name '*.wav' | while read line
do 
    lame --quiet --preset phone "$line" "$(basename "$line" .wav).mp3" &
    running="$(ps -u $USER | awk '{ if ($4 == "lame") print }' | wc -l)"
    while test "$running" = "$num_threads"
    do
        sleep 1
        running="$(ps -u $USER | awk '{ if ($4 == "lame") print }' | wc -l)"
    done
done

dreamchaser · 05-11-2015, 02:37 PM

Thanks a bunch for your replies. Switching from --execdir to --exec solved the problem.

Thanks metaschima for your elaborate answer. I was wondering how to multi-thread this task. Will try your suggestion next.