can't grep file

ust · 01-07-2008, 03:08 AM

when I use grep , it pop the below message ,

"bash: /bin/grep: Argument list too long"

I know this is because there are too many files in directory, except use the command "find" , how to fix it ? is it because the memory is not enough to list all files ? if yes , how to change the parameter to increase the memory ? thx

colucix · 01-07-2008, 04:13 AM

The "argument list too long" error is due to a kernel limitation relative to the number of characters that can be processed in the arguments list. I don't know if this limitation can be tweaked without recompiling the kernel, nor if it would be safe to do this. Anyway, you can also process one file at a time by the method you mentioned or by a simple for loop, e.g.

Code:

 for file in *.txt ; do grep "pattern" $file ; done

if some filename contains blank spaces, you have to set the IFS (input field separator) in order to avoid names splitted in two or more parts. A similar approach applies when you use find with the -print0 option to prevent this behavior.

ust · 01-07-2008, 04:29 AM

Quote:

Originally Posted by colucix

The "argument list too long" error is due to a kernel limitation relative to the number of characters that can be processed in the arguments list. I don't know if this limitation can be tweaked without recompiling the kernel, nor if it would be safe to do this. Anyway, you can also process one file at a time by the method you mentioned or by a simple for loop, e.g.

Code:

 for file in *.txt ; do grep "pattern" $file ; done

if some filename contains blank spaces, you have to set the IFS (input field separator) in order to avoid names splitted in two or more parts. A similar approach applies when you use find with the -print0 option to prevent this behavior.

thx reply , use "find" can solve my problem , however , other users have the same problem , I don't want to change the command they are used to apply , on the other hand , I hv many scripts need to change if use find , can advise other solution ? thx

colucix · 01-07-2008, 06:03 AM

I'm afraid there is not any other solution. The limit is set as a kernel parameter (ARG_MAX) and in linux the only way to increase this value is to recompile the kernel itself. But not advisable, since it is related to memory allocation during the execution of commands.

You can retrieve this value for your system by issuing

Code:

getconf ARG_MAX

or looking at /usr/include/linux/limits.h

You can find a good explanation of the ARG_MAX limit in the Coreutils FAQ, here and an even more detailed one, here.