-   Linux - Software (
-   -   Combine 10,000 small files into a big one (

Kunsheng 10-21-2009 08:54 AM

Combine 10,000 small files into a big one
I got around 10,000 files inside a folder, each of them contains around 20-30 lines.

Is there some quick way (probably bash command) to combine them into a big one ?


This number of files cause 'cat *' to fail and it reminds me 'argument list too long'.

A Perl program is working on combing it anyway, but just a little bit slow to me.

Lordandmaker 10-21-2009 09:08 AM

This sort of thing?

for i in `ls -1`; do
    cat $i >> newfile.txt

EDIT: If the perl script is slow, I can't see the above being any quicker.

*gets more coffee*

MensaWater 10-21-2009 09:16 AM

Assuming they're all in a single directory and there is nothing else there.
cd <directory>
find . -maxdepth 1 |grep -v <outputfile> |xargs cat >> <outputfile>

Of course you'd want to make sure where ever your outputfile is has enough space free to contain all your files output.

All times are GMT -5. The time now is 09:42 PM.