LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Software (http://www.linuxquestions.org/questions/linux-software-2/)
-   -   Combine 10,000 small files into a big one (http://www.linuxquestions.org/questions/linux-software-2/combine-10-000-small-files-into-a-big-one-763441/)

Kunsheng 10-21-2009 08:54 AM

Combine 10,000 small files into a big one
 
I got around 10,000 files inside a folder, each of them contains around 20-30 lines.

Is there some quick way (probably bash command) to combine them into a big one ?


PS:

This number of files cause 'cat *' to fail and it reminds me 'argument list too long'.

A Perl program is working on combing it anyway, but just a little bit slow to me.

Lordandmaker 10-21-2009 09:08 AM

This sort of thing?
Code:

for i in `ls -1`; do
    cat $i >> newfile.txt
done

EDIT: If the perl script is slow, I can't see the above being any quicker.

*gets more coffee*

MensaWater 10-21-2009 09:16 AM

Assuming they're all in a single directory and there is nothing else there.
cd <directory>
find . -maxdepth 1 |grep -v <outputfile> |xargs cat >> <outputfile>

Of course you'd want to make sure where ever your outputfile is has enough space free to contain all your files output.


All times are GMT -5. The time now is 01:46 PM.