-   Linux - General (
-   -   [quick] trying to split a large file but linux says it's to large (

steve51184 05-04-2008 11:40 PM

[quick] trying to split a large file but linux says it's to large
hi i'm trying to move servers and move all my data to the other server to i archived it up and it's a 90GB file but when i try to split it with a program like lxsplit i get the below error:


error opening input file: File too large

steve51184 05-04-2008 11:46 PM

just tried the split command and it gave this error:


split: ytes=2048m: invalid number of bytes

billymayday 05-04-2008 11:52 PM

I assume that's a typo on your part (split: ytes=2048m)

What happens if you try

split -b 2097152 infile

steve51184 05-05-2008 12:04 AM

it wasn't a typo that's just how the error was coming out with the "split –bytes=2048m file" command

that command you suggested is working but i added 3 more "0" so i get 2000mb files and not 2mb files but i have a question

i'm getting files like xaa, xab, xab etc and there will be about 50 of these but how do i join them back together?

billymayday 05-05-2008 12:09 AM

Should have been

split –-bytes=2048m file (2 dashes)

use cat to rejoin

steve51184 05-05-2008 12:14 AM

thanks for your help (again!) but there will be 50 randomly named files how do i join them with "cat" ?

billymayday 05-05-2008 12:23 AM

There's nothing random in the files, you can set the prefix and the suffix as per man split.

cat xaa xab xac > outfile

will work, but you'll probably want to script a loop for 50 files.

steve51184 05-05-2008 12:37 AM


Originally Posted by billymayday (Post 3142935)
you'll probably want to script a loop for 50 files.

hate to be a pain but how do i do that?

win32sux 05-05-2008 01:28 AM


Originally Posted by steve51184 (Post 3142946)
hate to be a pain but how do i do that?

If all the chunks' filenames start with the lowercase letter x and are exactly three characters long then an example of how to do it might be:

for i in $(ls x??); do cat $i >> example.tar.bz2; done
Of course, you'll probably have nothing but the chunks in the directory on the new box, and in such a case you wouldn't need to be specific at all:

for i in $(ls); do cat $i >> example.tar.bz2; done

billymayday 05-05-2008 01:32 AM

Thanks - my looping is crap or I would have done it in my previous post.

steve51184 05-05-2008 01:35 AM

thank you both very much!! when and split is done and i've moved them to my other server i'll try that loop command :)

win32sux 05-05-2008 01:52 AM

No problem. BTW, make sure you checksum your file. :)

steve51184 05-05-2008 01:59 AM

good idea.. how do i do that on linux? cksum filename right?

jschiwal 05-05-2008 02:30 AM

I once backed up my /home directory before installing SuSE 10.3. The resultant file would too large (30-40GB) for the vfat external drive I was backing up to. So I piped the output of tar through split and saved the slices on the external drive.

To restore, I mounted the drive, used cat and piped the output to the tar command. I didn't have to join the file first. I also used par2 to create parity files to enable the recovery of a damaged slice.

The examples near the end of the tar command has an example of moving a large directory using the -C option or a subshell on the right hand side of the pipe.

billymayday 05-05-2008 02:31 AM

I'm not sure anout all the option here. I use md5sum, but there's also sha1sum and I would guess more.

Just have a look at man md5sum for example.

Do this before the split and then after the rejoining on the other machine

All times are GMT -5. The time now is 09:50 AM.