LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 05-09-2015, 06:18 PM   #1
dreamchaser
LQ Newbie
 
Registered: May 2015
Posts: 2

Rep: Reputation: Disabled
Too many open files in system


Hi All,

Batch converting a large number of files from wav to mp3 with the following command eventually terminates with a "Too many open files in system" error:

"find /recordings/arrow82/monitor/2015/04 -name '*.wav' -execdir sh -c 'lame --quiet --preset phone '{}' '{}'.mp3' \;"

I increased the limits in /etc/security/limits.conf to:

root hard nofile 999999
root soft nofile 999999

...that helped for a while but there are now 200k+ files in the directory in which find is running.

Can the open files limit be increased further, or are there better ways of processing files in a directory tree?
 
Old 05-09-2015, 07:18 PM   #2
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,786

Rep: Reputation: 2216Reputation: 2216Reputation: 2216Reputation: 2216Reputation: 2216Reputation: 2216Reputation: 2216Reputation: 2216Reputation: 2216Reputation: 2216Reputation: 2216
The problem is the find command trying to retain a handle on the working directory for each file that is used in an execdir. You will need to either run separate find commands for different subsets of your .wav files, or else use "-exec" instead of "-execdir".
 
Old 05-09-2015, 07:22 PM   #3
metaschima
Senior Member
 
Registered: Dec 2013
Distribution: Slackware
Posts: 1,982

Rep: Reputation: 492Reputation: 492Reputation: 492Reputation: 492Reputation: 492
What you probably want is to convert the files a set number at a time.

One file at a time:
Code:
find /recordings/arrow82/monitor/2015/04 -type f -name '*.wav' | while read line
do 
    lame --quiet --preset phone "$line" "$(basename "$line" .wav).mp3"
done
You can run this a few at a time per directory.

I'll have to think of a clever way to run more than one at a time.

EDIT:

Code:
num_threads=4

find /recordings/arrow82/monitor/2015/04 -type f -name '*.wav' | while read line
do 
    lame --quiet --preset phone "$line" "$(basename "$line" .wav).mp3" &
    running="$(ps -u $USER | awk '{ if ($4 == "lame") print }' | wc -l)"
    while test "$running" = "$num_threads"
    do
        sleep 1
        running="$(ps -u $USER | awk '{ if ($4 == "lame") print }' | wc -l)"
    done
done

Last edited by metaschima; 05-09-2015 at 07:51 PM.
 
Old 05-11-2015, 02:37 PM   #4
dreamchaser
LQ Newbie
 
Registered: May 2015
Posts: 2

Original Poster
Rep: Reputation: Disabled
Thanks a bunch for your replies. Switching from --execdir to --exec solved the problem.

Thanks metaschima for your elaborate answer. I was wondering how to multi-thread this task. Will try your suggestion next.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Linpus Fedora after saving files to file system, CAN'T open FILE SYSTEM from desktop! remiwa Fedora 2 01-07-2009 07:28 AM
Too many open files in system DanaBrown Linux - Newbie 2 07-04-2008 02:41 PM
Too many open files in system ipconfig Linux - Newbie 5 12-16-2003 05:14 AM
Too many open files in system jayakrishnan Linux - General 5 03-28-2003 09:58 PM
Too many open files on system error?! sarin Linux - General 4 07-23-2002 06:49 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 03:24 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration