LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 06-27-2011, 01:56 AM   #1
muby
LQ Newbie
 
Registered: Oct 2007
Posts: 28

Rep: Reputation: 15
Exclamation ls, cp & tar for 80K+ files


Hi Everybody,

I need to perform ls, cp & tar in a directory that contains 80k text files, but i understand that there are limitation for files 70000 files and above.

I would like to know how can i perform such command e.g "ls -l | wc -l" with so many files.

thanks in advance.
 
Old 06-27-2011, 02:44 AM   #2
John VV
LQ Muse
 
Registered: Aug 2005
Location: A2 area Mi.
Posts: 17,093

Rep: Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474
this sounds like homework
did you try ? and IS there that 70000 max ?
That is did it not tar up the last 10000

70000 sounds a bit "odd " 65536 however
 
Old 06-27-2011, 03:00 AM   #3
muby
LQ Newbie
 
Registered: Oct 2007
Posts: 28

Original Poster
Rep: Reputation: 15
Exclamation

65536 could be the actual limit, but i only tried tar on 64000 (successful) and 71000 (failed).

I need to copy those files after tar to another server & keep a backup.

i guess eventually need to divide the files to 2 tar files each 40K files. but i have no clue how to do this.
 
Old 06-27-2011, 03:15 AM   #4
Wim Sturkenboom
Senior Member
 
Registered: Jan 2005
Location: Roodepoort, South Africa
Distribution: Slackware 10.1/10.2/12, Ubuntu 12.04, Crunchbang Statler
Posts: 3,786

Rep: Reputation: 282Reputation: 282Reputation: 282
just some ideas that I can come up with:

filter by extension (if they all end with e.g. 'txt', it will not work)
filter by first letter of filename ('*a*', 'b*' etc)

or maybe -M option (never used it myself) or the --exclude option
 
Old 06-27-2011, 05:36 AM   #5
MTK358
LQ 5k Club
 
Registered: Sep 2009
Posts: 6,443
Blog Entries: 3

Rep: Reputation: 721Reputation: 721Reputation: 721Reputation: 721Reputation: 721Reputation: 721Reputation: 721
Quote:
Originally Posted by Wim Sturkenboom View Post
just some ideas that I can come up with:

filter by extension (if they all end with e.g. 'txt', it will not work)
filter by first letter of filename ('*a*', 'b*' etc)

or maybe -M option (never used it myself) or the --exclude option
This is what I would do, there's no chance of the number of files going over the limit:

Code:
tmpdir = "/tmp/${RANDOM}-$(date +%s)"
mkdir "$tmpdir"
ls -1 > "$tmpdir/files" # the option to "ls" is the number one, not the lowercase letter "L"
split --lines=64000 "$tmpdir/files" part

for part in "$tmpdir"/part*
do
    # do stuff. "$part" will contain the path to a file with a newline-separated list of filenames
done

rm "$tmpdir"/*
rmdir "$tmpdir"

Last edited by MTK358; 06-27-2011 at 05:38 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
a tough question 4 u, problem in extracting tar & tar.gz files p_garg Linux - General 5 11-08-2010 11:02 AM
More tar issues - listing files in a single tar from a multivolume borks Thymox Linux - Software 4 09-23-2009 07:48 PM
How to install tar.gz & tar.bz2 file on Linux min Krishnendu Linux Mint 2 07-16-2009 06:52 AM
BackUp & Restore with TAR (.tar / .tar.gz / .tar.bz2 / tar.Z) asgarcymed Linux - General 5 12-31-2006 02:53 AM
.rpms, .tar.gz, .tgz, .src.rpm, & .tar.bz2 whoots Mandriva 10 10-18-2003 12:08 PM


All times are GMT -5. The time now is 03:11 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration