Combine 10,000 small files into a big one

Kunsheng · 10-21-2009, 08:54 AM

I got around 10,000 files inside a folder, each of them contains around 20-30 lines.

Is there some quick way (probably bash command) to combine them into a big one ?

PS:

This number of files cause 'cat *' to fail and it reminds me 'argument list too long'.

A Perl program is working on combing it anyway, but just a little bit slow to me.

Lordandmaker · 10-21-2009, 09:08 AM

This sort of thing?

Code:

for i in `ls -1`; do
    cat $i >> newfile.txt
done

EDIT: If the perl script is slow, I can't see the above being any quicker.

*gets more coffee*

MensaWater · 10-21-2009, 09:16 AM

Assuming they're all in a single directory and there is nothing else there.
cd <directory>
find . -maxdepth 1 |grep -v <outputfile> |xargs cat >> <outputfile>

Of course you'd want to make sure where ever your outputfile is has enough space free to contain all your files output.