LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware
User Name
Password
Slackware This Forum is for the discussion of Slackware Linux.

Notices

Reply
 
Search this Thread
Old 07-06-2002, 09:49 PM   #1
jaysan
LQ Newbie
 
Registered: May 2001
Distribution: Slackware Linux 9.1
Posts: 23

Rep: Reputation: 15
file size limit


hello,

i have just installed slackware 8.1 with reiserfs.
i tried to backup data from our sun workstations using cpio but
there seems to be a 2Gb limit on the filesize.

is there a max file size limit for cpio?
is there a way to get around this limit?

the kernel installed is 2.4.18.

I also experienced this problem with slackware 8.0,
i installed the 2.2.19 kernel using ext2fs. I was hoping that
upgrading to slackware 8.1 using the 2.4.18 kernel and reiserfs would help solve this problem.

please help...

TIA
 
Old 07-06-2002, 11:16 PM   #2
Excalibur
Senior Member
 
Registered: Jun 2002
Location: Northern VA, USA
Distribution: Ubuntu
Posts: 1,180

Rep: Reputation: 45
I have experienced similar problems. Even though a file system is suppose to support larger file sizes than 2 GB, most programs do not. And even if you find an app that will, most every operation you attempt with it will also fail. Your mileage may vary.

The only solution I was able to work with consistently and reliably was to make smaller files.

Cpio may very well have a file size limitation imposed through bash if your output is to a file or it may have a limitation within if the file to be backed up is greater than 2 GB. I am sure I have been able to far exceed 2 GB on total output using tar to a device (not a file), like a hard disk, DVD or tape. My daily backup on my workstation exceeds 3 GB on DVD+RW using tar piped to sdd to handle the write operation. But tar is purely sequential, it is a aging process to restore a single file from such a backup. You might want to investigate a program called "taper". I believe it allows file selections through an index list if I recall. I think the output was designed for tape drives though. If you have the disk space and can set it up, like on a network, you may want to look at rsync. On rsync though, I do not know if a single file over 2 GB would be a problem or not.

I have not used cpio direct to a device but I think it supports tape drives so you could attempt the following. On for example a 20 gbyte hard disk for backups, create 4 primary partitions of 5 GB each. Then perform your backup using the device name directly. No file system. Reference it like /dev/hdd1, hdd2, hdd3, hdd4. Modify for your needs as required.

If you are interested in a backup medium faster than tape, you may want to investigate DVD+RW. Last night my system backed up 3.9 GB in 20 minutes. (200 MB/minute.) The capacity without compression is 4.7 GB.

Good luck.
 
Old 07-11-2002, 06:28 PM   #3
jaysan
LQ Newbie
 
Registered: May 2001
Distribution: Slackware Linux 9.1
Posts: 23

Original Poster
Rep: Reputation: 15
filesize limit

Excalubur,

thank you very much for your suggestion. I tried using tar but it did not work. It also stopped at 2GB.

With regards to the 5 partitions that you suggested. The backup drives are physically not at the said machine. It was mounted from our NT based stations using smbmount. Could it be that smbmount has a problem with filesize exceeding 2GB?

I wish we had a tape drive but we cannot afford it.

Anyway, i also have a red hat linux machine with 2.4.2 kernel using ext2fs. It can tar files at more than 2GB.
 
Old 07-11-2002, 10:03 PM   #4
zelgadis
Member
 
Registered: Apr 2002
Location: Venezuela
Distribution: slackware 8.0
Posts: 67

Rep: Reputation: 15
Very complete information in:

http://www.suse.de/~aj/linux_lfs.html

and more links about this and other subjects in:

http://loll.sourceforge.net/linux/links/
 
Old 07-11-2002, 11:17 PM   #5
Excalibur
Senior Member
 
Registered: Jun 2002
Location: Northern VA, USA
Distribution: Ubuntu
Posts: 1,180

Rep: Reputation: 45
Zelgadis, thank you for the suse.de link. It was most informative. But it appears that we are still not quite there as yet.

I was able to duplicate the 2 GB limit using cpio on local source files to local output. However, when I used tar to perform the same backup it created a file that the "ls" command wouldn't even display in the directory. And "rm" would not delete it. I had to use the tar command again to truncate to an acceptable size. Then delete it.

I can only recommend for the backup limitation issue using cpio; break it up into smaller files sizes, or performing data only backups. You can also consider piping the output through gzip to compress it. But that is only a limited workaround until the file output reaches the 2 GB limit again because gzip has the same limitation. Consider also the network usage for complete backups and time constraints for these massive transfer operations. For a Linux box data only backup consider the directories; /etc, /home, and /root. I also include /var/spool/mail and /var/spool/mqueue if it is a mail server and the apache files if a web server.

In regards to my suggestion on the hard disk partitions, I would think the disk physically needs to be where the tar/cpio command is executed would be best. But I do not see why it couldn't be remote on a Linux box, just use the HOSTNAME:/dev/hddx to reference it like a tape drive would be. If you are backing up to a destination located on a NT workstation that is using NTFS it should not have a 2/4 GB limit. If other, Win2K Pro, using FAT32 then high possibility of 2/4 GB limit.

Good luck on your endeavor.
 
Old 07-12-2002, 03:43 AM   #6
jaysan
LQ Newbie
 
Registered: May 2001
Distribution: Slackware Linux 9.1
Posts: 23

Original Poster
Rep: Reputation: 15
filesize limit

hi,

i tried to tar it on the local drive, it reached 2.3GB when i stopped it because my drive was almost empty.

could it be that smbmount is the problem?

zelgadis,

thanks for the link, very informative, though i think i need to read it a second time... am still a newbie...
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
samba file size limit? babag Linux - Networking 2 10-12-2005 02:26 PM
File Size limit? compu73rg33k Linux - Software 1 02-27-2005 10:11 PM
limit file size ust Linux - General 2 02-24-2005 07:33 AM
File Size Limit?? Manuel-H Linux - Software 1 04-16-2004 03:23 AM
File size limit petway56 Linux - General 4 05-07-2003 11:19 PM


All times are GMT -5. The time now is 08:13 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration