Linux - EnterpriseThis forum is for all items relating to using Linux in the Enterprise.
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Introduction to Linux - A Hands on Guide
This guide was created as an overview of the Linux Operating System, geared toward new users as an exploration tour and getting started guide, with exercises at the end of each chapter.
For more advanced trainees it can be a desktop reference, and a collection of the base knowledge needed to proceed with system and network administration. This book contains many real life examples derived from the author's experience as a Linux system and network administrator, trainer and consultant. They hope these examples will help you to get a better understanding of the Linux system and that you feel encouraged to try out things on your own.
Click Here to receive this Complete Guide absolutely free.
I have an LTO3 400GB/800GB backup tape unit connected to my 1.2TB NAS server. To date, I only have 110GB of data to backup. AT THE VERY MINIMUM, I should be getting around 30MB/sec backup speed however, I'm doing a simple afio to tape and getting a painfully slow 2.5MB/s. My older DLT tapes were much faster then this!
How in the heck can I test where the bottleneck is? I have no idea where to start. Any hints would be appreciated.
What software is being used to backup this data? How or where are you getting this number from? What type of data? What type of network? How do you have this LTO3 connected, via tape library, standalone drive, etc? More or better details on your setup gives us more insight.
I've dealt with LTO3 drives in a 200 plus server environment, with 12 total drives in a SL8500 Storagetek library with a total of around 500TB of backups at any given time. I probably averaged around 20 to 45MB/s running at max utilization.
The data is mostly made up of compressed Acronis TrueImage server backup images. As a test, I only have 1 large 110GB image going to a directly connected LTO3 standalone drive attached to its own Adaptec 29320ALP controller card.
I've tried messing with various, drive, kernel, afio, and tar buffer settings and only get at max a 5 second difference in write time during testing. For a test I have a directory with 211MB in it that takes 2min15sec to put to tape. 211MB / 135sec = an amazing 1.5MB/s!
The Scsi Ultra320 card is set to the default settings, but the tape drive crashes if I set it to anything else other then full on 320MB/s.
I guess I'm not understanding why your using 'find' and piping it to afio to backup files. Have you tried just specifying a file to backup directly? I'm not sure what's all in this /shares directory as you're first printing 110GB of files before it pipes it to the tape for backup and you have but you also have to consider initilizing the tape in the drive, rewind time, whatever else is done prior before it's actually writing to tape.
What's the time frame it takes without piping to afio? Knock that time off the total time. Also, you may consider trying another type of software to backup to test, as in dump or amanda, etc. And if this shares directory consists of a lot of tiny files, that can also take a little longer to write to tape instead of larger files, especially since it's only 110GB of data. You should really try larger files or a larger backup to test.
I'm piping find to afio because it's an old cpio habit. The 'vv' is used with the '-L /tmp/afio_errors' as a crude index so I can find files quickly when I need to restore them. I'm aware that printing the files like this is slower, however it shouldn't be the difference between a backup taking 2 hours and a backup taking 11 hours. My 110GB backup takes between 11 hours to 12 hours to write to tape!
The shares directory is only one file at this current time. That one file is 110GB in size.
I've used tar without the -Z flag to test and get the EXACT same results. It takes 2 minutes and 15 seconds to write 211MB to tape and close to 12 hours for the entire backup.
Right now I can't use 'dump' because I use the default Suse filesystem (ReiserFS). I have considered changing it to ext2 or ext3 to test dump, but just haven't yet.
Amanda to my knowledge is a wrapper for dump and\or cpio which wouldn't change my current situation. I also don't want\need all the extra tape management overhead. My backup specialist is not a Linux expert.
Thanks again for the replies. I've been trying options for well over a week and no matter what I do, it changes the throughput very little. I've checked and rechecked my hardware and it all seems okay, so I'm really stumped. These same backup commands on my old DLT on a different server would average around 7MB/s. That would be a dream if I could even get that throughput at this point!
I have two clone NAS units so I took the second one and put Ubuntu using ext2 FS on it for further testing. The results are the same. It is taking 11 hours to backup up to tape. This is getting beyond frustrating honestly. I've got a lot of $$ ties up in these things and as of now, they are just about useless for my intended purpose.
I keep having people tell me that large files should transfer faster then small ones. My testing is showing the exact opposite. I'm backing-up my etc and shares directory. The etc directory is a ton of small files and my shres directory is currently one huge image that is 110GB is size. When I break them down and back them up one at a time, I get a contiguous stream to the tape when backing up the etc directory. When I back up the image file, it writes for about 3 seconds then stops for about 15 to 20 seconds then writes for another 3 seconds,...on and on and on for over 11 hours.
Are these results typical? If not, does anyone have an idea what could be going on? I'm not even real sure what to check next.
Well, isn't this interesting,... in Windows Server 2003 the very same backup gets 58 MB/s. Apparently something is messed up somewhere in the kernel or drivers or something, but since I can't figure it out, I guess,... **GULP!!**,...my NAS unit will run Windows from now on!
Did you contact Certance? I had the same problem with my drive and the solution was to use a different SCSI controller. Don't ask me why, but apparently the certance drives really don't like onboard controllers... the old controller worked fine under Linux with everything but this particular tape drive.
I ended up getting a new controller card as well. I had a SCSI 320 card in the server. When I put it back down to 160 it wouldn't work. I switched it out to a straight 160 card and now it screams! Sounds like we had the same problem and found the same solution.