Split a large file to download on a windows machine
Linux - NewbieThis Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place!
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
This has created a tar file for each chunk. If you don't have room to do them all at once, then you can script the loop to tar up one chunk, scp it off to some other storage, delete it, then do the next.
Tried but for some reason near the end I got a bunch of these errors, did an ls and was able to verify that these files are there:
tar: xyz55823.jpg*: Cannot stat: No such file or directory
tar: xyz55824.jpg*: Cannot stat: No such file or directory
tar: xyz55825.jpg*: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors trackpad@trackpads.com [~/www/path/to/2658]#
Tried but for some reason near the end I got a bunch of these errors, did an ls and was able to verify that these files are there:
tar: xyz55823.jpg*: Cannot stat: No such file or directory
tar: xyz55824.jpg*: Cannot stat: No such file or directory
tar: xyz55825.jpg*: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors trackpad@trackpads.com [~/www/path/to/2658]#
You may need to run fsck on the partition when it is not mounted.
Firstly, from your title, I'm not certain which machine is Windows. The client or the server. Since is seems you ran tar on the server, is sounds like the client is Windows.
Is it the tar command that failed before it got to the split command. You may simply have some bad files or files that were altered after you started tar.
Tar does have a volume size option, which would allow you to create separate files.
Another option is to pipe the output of tar. For example:
ssh user@server tar -C <directory> czf - . | cat >gallery.tar.gz
You could instead use tar to replicate the files themselves:
ssh user@server tar -C <directory> czf - . | tar -C <restore_directory> xzvf - >logfile
If the client is a windows machine, you could run a live linux distro or Cygwin to run the ssh and tar commands.
The problem you are having is that there are too many files in the directory rather than its size. For example, trying "ls *.jpg" may cause an out of memory error. This is because the wildcard is expanded and sorted by the shell before the command is executed. Using ftp, a list of all of the files in the directory may be produced so it can be sorted as well. Sorting is expensive time wise.
One thing that is often done in the case where there are 10s of thousands of files or more in a directory is to use the find command instead of ls, and to limit the number of arguments to a command using xargs to handle the list produced by find. If you use tar without wild card arguments, you probably won't have the problem I mentioned. Files are added as they are found.
I have not tried smallpond's solution but it looks ok to me. One thing I would change, though, would be
ls > allfiles
to
ls -1 > allfiles
This way you can assure every tar command will archive 1000 files.
As for the size issue of the chunks, maybe you have some very long filenames? Or it ran out of output filenames and put the rest in the last chunk? But this should have given an error message.
Try this to mimic the split command
Finally, the file not found issue. Here is a very far fetched idea:
My guess would be that,
- you put "double-quotes" around "$i" in smallpond's example and
- your 'ls' command is really aliased to 'ls -F' and
- the files in question somehow got falsly execute permissions assigned.
The output of
Code:
alias ls
ls -l xyz55823.jpg
will show if my far fetched assumption holds any truth
Last edited by crts; 04-16-2011 at 12:20 PM.
Reason: typos
I got those downloaded but now have an even bigger single files archive.
I am moving from a linux host to a windows host. The file I need to download is my cpanel backup which is about 200gb. It is a tar.gz file. Can I split that into increments of say 500mb and then reassemble them on my windows box?
I got those downloaded but now have an even bigger single files archive.
I am moving from a linux host to a windows host. The file I need to download is my cpanel backup which is about 200gb. It is a tar.gz file. Can I split that into increments of say 500mb and then reassemble them on my windows box?
Thanks again,
-Jason
Hi,
how did you solve the previous problems you encountered, i.e. the '.jpg*: Cannot stat: No such file or directory' issue? Some feedback would be nice for others who might stumble upon the same problem.
the description it sounds promising I have never used this tool before. So if you want to try then make a backup first and do some testruns with smaller dummy-files.
Hope this helps.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.