LinuxQuestions.org
Did you know LQ has a Linux Hardware Compatibility List?
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices



Reply
 
Search this Thread
Old 10-21-2009, 09:54 AM   #1
Kunsheng
Member
 
Registered: Mar 2009
Posts: 82

Rep: Reputation: 16
Combine 10,000 small files into a big one


I got around 10,000 files inside a folder, each of them contains around 20-30 lines.

Is there some quick way (probably bash command) to combine them into a big one ?


PS:

This number of files cause 'cat *' to fail and it reminds me 'argument list too long'.

A Perl program is working on combing it anyway, but just a little bit slow to me.
 
Old 10-21-2009, 10:08 AM   #2
Lordandmaker
Member
 
Registered: Sep 2005
Location: London, UK
Distribution: Debian
Posts: 258

Rep: Reputation: 38
This sort of thing?
Code:
for i in `ls -1`; do
    cat $i >> newfile.txt
done
EDIT: If the perl script is slow, I can't see the above being any quicker.

*gets more coffee*

Last edited by Lordandmaker; 10-21-2009 at 10:09 AM. Reason: noticed the OP's last line.
 
Old 10-21-2009, 10:16 AM   #3
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,028
Blog Entries: 5

Rep: Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791
Assuming they're all in a single directory and there is nothing else there.
cd <directory>
find . -maxdepth 1 |grep -v <outputfile> |xargs cat >> <outputfile>

Of course you'd want to make sure where ever your outputfile is has enough space free to contain all your files output.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
howto combine pdf files JUbuntu Linux - Newbie 2 04-29-2008 09:15 PM
combine shadow files packets Linux - Newbie 2 04-18-2008 06:14 AM
LXer: SugarCRM Announces 1,000 Customers and 1,000,000 Open Source Downloads as Momentum for Open Source Applications Grows LXer Syndicated Linux News 0 12-19-2006 06:33 AM
1,000,000,000 PCs by 2010 masand Linux - News 4 11-01-2004 02:55 AM
Best way to combine files? JockVSJock Programming 4 04-25-2004 08:38 PM


All times are GMT -5. The time now is 09:00 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration