LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 05-04-2008, 11:40 PM   #1
steve51184
Member
 
Registered: Dec 2006
Posts: 372

Rep: Reputation: 30
[quick] trying to split a large file but linux says it's to large


hi i'm trying to move servers and move all my data to the other server to i archived it up and it's a 90GB file but when i try to split it with a program like lxsplit i get the below error:

Quote:
error opening input file: File too large
 
Old 05-04-2008, 11:46 PM   #2
steve51184
Member
 
Registered: Dec 2006
Posts: 372

Original Poster
Rep: Reputation: 30
just tried the split command and it gave this error:

Quote:
split: ytes=2048m: invalid number of bytes
 
Old 05-04-2008, 11:52 PM   #3
billymayday
Guru
 
Registered: Mar 2006
Location: Sydney, Australia
Distribution: Fedora, CentOS, OpenSuse, Slack, Gentoo, Debian, Arch, PCBSD
Posts: 6,678

Rep: Reputation: 122Reputation: 122
I assume that's a typo on your part (split: ytes=2048m)

What happens if you try

split -b 2097152 infile
 
Old 05-05-2008, 12:04 AM   #4
steve51184
Member
 
Registered: Dec 2006
Posts: 372

Original Poster
Rep: Reputation: 30
it wasn't a typo that's just how the error was coming out with the "split –bytes=2048m file" command

that command you suggested is working but i added 3 more "0" so i get 2000mb files and not 2mb files but i have a question

i'm getting files like xaa, xab, xab etc and there will be about 50 of these but how do i join them back together?

Last edited by steve51184; 05-05-2008 at 12:08 AM.
 
Old 05-05-2008, 12:09 AM   #5
billymayday
Guru
 
Registered: Mar 2006
Location: Sydney, Australia
Distribution: Fedora, CentOS, OpenSuse, Slack, Gentoo, Debian, Arch, PCBSD
Posts: 6,678

Rep: Reputation: 122Reputation: 122
Should have been

split –-bytes=2048m file (2 dashes)


use cat to rejoin
 
Old 05-05-2008, 12:14 AM   #6
steve51184
Member
 
Registered: Dec 2006
Posts: 372

Original Poster
Rep: Reputation: 30
thanks for your help (again!) but there will be 50 randomly named files how do i join them with "cat" ?
 
Old 05-05-2008, 12:23 AM   #7
billymayday
Guru
 
Registered: Mar 2006
Location: Sydney, Australia
Distribution: Fedora, CentOS, OpenSuse, Slack, Gentoo, Debian, Arch, PCBSD
Posts: 6,678

Rep: Reputation: 122Reputation: 122
There's nothing random in the files, you can set the prefix and the suffix as per man split.

cat xaa xab xac > outfile

will work, but you'll probably want to script a loop for 50 files.
 
Old 05-05-2008, 12:37 AM   #8
steve51184
Member
 
Registered: Dec 2006
Posts: 372

Original Poster
Rep: Reputation: 30
Quote:
Originally Posted by billymayday View Post
you'll probably want to script a loop for 50 files.
hate to be a pain but how do i do that?
 
Old 05-05-2008, 01:28 AM   #9
win32sux
Guru
 
Registered: Jul 2003
Location: Los Angeles
Distribution: Ubuntu
Posts: 9,870

Rep: Reputation: 371Reputation: 371Reputation: 371Reputation: 371
Quote:
Originally Posted by steve51184 View Post
hate to be a pain but how do i do that?
If all the chunks' filenames start with the lowercase letter x and are exactly three characters long then an example of how to do it might be:
Code:
for i in $(ls x??); do cat $i >> example.tar.bz2; done
Of course, you'll probably have nothing but the chunks in the directory on the new box, and in such a case you wouldn't need to be specific at all:
Code:
for i in $(ls); do cat $i >> example.tar.bz2; done

Last edited by win32sux; 05-05-2008 at 01:35 AM.
 
Old 05-05-2008, 01:32 AM   #10
billymayday
Guru
 
Registered: Mar 2006
Location: Sydney, Australia
Distribution: Fedora, CentOS, OpenSuse, Slack, Gentoo, Debian, Arch, PCBSD
Posts: 6,678

Rep: Reputation: 122Reputation: 122
Thanks - my looping is crap or I would have done it in my previous post.
 
Old 05-05-2008, 01:35 AM   #11
steve51184
Member
 
Registered: Dec 2006
Posts: 372

Original Poster
Rep: Reputation: 30
thank you both very much!! when and split is done and i've moved them to my other server i'll try that loop command
 
Old 05-05-2008, 01:52 AM   #12
win32sux
Guru
 
Registered: Jul 2003
Location: Los Angeles
Distribution: Ubuntu
Posts: 9,870

Rep: Reputation: 371Reputation: 371Reputation: 371Reputation: 371
No problem. BTW, make sure you checksum your file.
 
Old 05-05-2008, 01:59 AM   #13
steve51184
Member
 
Registered: Dec 2006
Posts: 372

Original Poster
Rep: Reputation: 30
good idea.. how do i do that on linux? cksum filename right?
 
Old 05-05-2008, 02:30 AM   #14
jschiwal
Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654
I once backed up my /home directory before installing SuSE 10.3. The resultant file would too large (30-40GB) for the vfat external drive I was backing up to. So I piped the output of tar through split and saved the slices on the external drive.

To restore, I mounted the drive, used cat and piped the output to the tar command. I didn't have to join the file first. I also used par2 to create parity files to enable the recovery of a damaged slice.

The examples near the end of the tar command has an example of moving a large directory using the -C option or a subshell on the right hand side of the pipe.
 
Old 05-05-2008, 02:31 AM   #15
billymayday
Guru
 
Registered: Mar 2006
Location: Sydney, Australia
Distribution: Fedora, CentOS, OpenSuse, Slack, Gentoo, Debian, Arch, PCBSD
Posts: 6,678

Rep: Reputation: 122Reputation: 122
I'm not sure anout all the option here. I use md5sum, but there's also sha1sum and I would guess more.

Just have a look at man md5sum for example.

Do this before the split and then after the rejoining on the other machine
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
how do I split large file by string? khairil Programming 5 04-28-2008 10:37 PM
Split large file in several files using scripting (awk etc.) chipix Programming 14 10-29-2007 11:16 AM
Split a large file and get the names of output files using Perl Sherlock Programming 25 02-02-2007 12:43 PM
Split large file into multiples jdozarchuk Linux - Newbie 1 11-04-2004 09:42 AM
split a large mpeg file into two zstingx Linux - General 3 11-06-2003 06:26 PM


All times are GMT -5. The time now is 05:31 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration