LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Enterprise (http://www.linuxquestions.org/questions/linux-enterprise-47/)
-   -   LTO Backup Speed (http://www.linuxquestions.org/questions/linux-enterprise-47/lto-backup-speed-489488/)

ghight 10-04-2006 03:57 PM

LTO Backup Speed
 
I have an LTO3 400GB/800GB backup tape unit connected to my 1.2TB NAS server. To date, I only have 110GB of data to backup. AT THE VERY MINIMUM, I should be getting around 30MB/sec backup speed however, I'm doing a simple afio to tape and getting a painfully slow 2.5MB/s. My older DLT tapes were much faster then this!

How in the heck can I test where the bottleneck is? I have no idea where to start. Any hints would be appreciated.

Thanks in advance
ghight

trickykid 10-05-2006 06:26 PM

What software is being used to backup this data? How or where are you getting this number from? What type of data? What type of network? How do you have this LTO3 connected, via tape library, standalone drive, etc? More or better details on your setup gives us more insight.

I've dealt with LTO3 drives in a 200 plus server environment, with 12 total drives in a SL8500 Storagetek library with a total of around 500TB of backups at any given time. I probably averaged around 20 to 45MB/s running at max utilization.

ghight 10-05-2006 08:16 PM

The software being used is "afio" on Suse 10.1. The business end of the script is simply:

Code:

find /shares/ /etc/ -depth -print0 | afio -oxvv -b 128k -0a -L /tmp/afio_errors /dev/st0
I'm using stinit to setup my drive at boot. the stinit.def contents are as follows:
Code:

manufacturer="CERTANCE" model = "ULTRIUM" revision = "1856" {
scsi2logical=1
can-bsr=1
auto-lock=0
two-fms=0
drive-buffering=1
buffer-writes
read-ahead=1
async-writes=1
can-partitions=1
fast-mteom=1
timeout=180
long-timeout=14400
mode1 blocksize=0 compression=1 # 800 GB, native
#mode2 blocksize=0 compression=0 # 400 GB, LTO
}

The drive has hardwre compression on, drive buffering on, and variable block-mode rather then fixed block mode. These seem to give me the best numbers even though they are still dog slow.

tapeinfo output:
Code:

Product Type: Tape Drive
Vendor ID: 'CERTANCE'
Product ID: 'ULTRIUM 3      '
Revision: '1856'
Attached Changer: No
SerialNumber: 'JD00A94    '
MinBlock:1
MaxBlock:16777215
SCSI ID: 6
SCSI LUN: 0
Ready: yes
BufferedMode: yes
Medium Type: Not Loaded
Density Code: 0x44
BlockSize: 0
DataCompEnabled: yes
DataCompCapable: yes
DataDeCompEnabled: yes
CompType: 0x1
DeCompType: 0x1
BOP: yes
Block Position: 0

The data is mostly made up of compressed Acronis TrueImage server backup images. As a test, I only have 1 large 110GB image going to a directly connected LTO3 standalone drive attached to its own Adaptec 29320ALP controller card.

I've tried messing with various, drive, kernel, afio, and tar buffer settings and only get at max a 5 second difference in write time during testing. For a test I have a directory with 211MB in it that takes 2min15sec to put to tape. 211MB / 135sec = an amazing 1.5MB/s!

The Scsi Ultra320 card is set to the default settings, but the tape drive crashes if I set it to anything else other then full on 320MB/s.

Any ideas would be GREATLY appreciated.

trickykid 10-05-2006 10:00 PM

I guess I'm not understanding why your using 'find' and piping it to afio to backup files. Have you tried just specifying a file to backup directly? I'm not sure what's all in this /shares directory as you're first printing 110GB of files before it pipes it to the tape for backup and you have but you also have to consider initilizing the tape in the drive, rewind time, whatever else is done prior before it's actually writing to tape.

What's the time frame it takes without piping to afio? Knock that time off the total time. Also, you may consider trying another type of software to backup to test, as in dump or amanda, etc. And if this shares directory consists of a lot of tiny files, that can also take a little longer to write to tape instead of larger files, especially since it's only 110GB of data. You should really try larger files or a larger backup to test.

trickykid 10-05-2006 10:08 PM

I did a test of your find command on one of my systems, I chose my /usr directory which is around 1GB in size, took it 1 minute just to print the whole directory and files..

ghight 10-06-2006 08:38 AM

Tricky, thanks for the replies.

I'm piping find to afio because it's an old cpio habit. The 'vv' is used with the '-L /tmp/afio_errors' as a crude index so I can find files quickly when I need to restore them. I'm aware that printing the files like this is slower, however it shouldn't be the difference between a backup taking 2 hours and a backup taking 11 hours. My 110GB backup takes between 11 hours to 12 hours to write to tape!

110,000MB / 36,900 seconds = 2.7MB/s! (aka, SLOW!)

The shares directory is only one file at this current time. That one file is 110GB in size.

I've used tar without the -Z flag to test and get the EXACT same results. It takes 2 minutes and 15 seconds to write 211MB to tape and close to 12 hours for the entire backup.

Right now I can't use 'dump' because I use the default Suse filesystem (ReiserFS). I have considered changing it to ext2 or ext3 to test dump, but just haven't yet.

Amanda to my knowledge is a wrapper for dump and\or cpio which wouldn't change my current situation. I also don't want\need all the extra tape management overhead. My backup specialist is not a Linux expert.

Thanks again for the replies. I've been trying options for well over a week and no matter what I do, it changes the throughput very little. I've checked and rechecked my hardware and it all seems okay, so I'm really stumped. These same backup commands on my old DLT on a different server would average around 7MB/s. That would be a dream if I could even get that throughput at this point!

ghight 10-20-2006 04:09 PM

Alright, lets move this back to the top.

I have two clone NAS units so I took the second one and put Ubuntu using ext2 FS on it for further testing. The results are the same. It is taking 11 hours to backup up to tape. This is getting beyond frustrating honestly. I've got a lot of $$ ties up in these things and as of now, they are just about useless for my intended purpose.

I keep having people tell me that large files should transfer faster then small ones. My testing is showing the exact opposite. I'm backing-up my etc and shares directory. The etc directory is a ton of small files and my shres directory is currently one huge image that is 110GB is size. When I break them down and back them up one at a time, I get a contiguous stream to the tape when backing up the etc directory. When I back up the image file, it writes for about 3 seconds then stops for about 15 to 20 seconds then writes for another 3 seconds,...on and on and on for over 11 hours.

Are these results typical? If not, does anyone have an idea what could be going on? I'm not even real sure what to check next.

ghight 10-25-2006 03:58 PM

Well, isn't this interesting,... in Windows Server 2003 the very same backup gets 58 MB/s. Apparently something is messed up somewhere in the kernel or drivers or something, but since I can't figure it out, I guess,... **GULP!!**,...my NAS unit will run Windows from now on!

Man I seriously hate it when that happens!

cornel 05-25-2007 01:16 PM

Did you contact Certance? I had the same problem with my drive and the solution was to use a different SCSI controller. Don't ask me why, but apparently the certance drives really don't like onboard controllers... the old controller worked fine under Linux with everything but this particular tape drive. :confused:

Cornel

ghight 05-25-2007 01:45 PM

I ended up getting a new controller card as well. I had a SCSI 320 card in the server. When I put it back down to 160 it wouldn't work. I switched it out to a straight 160 card and now it screams! Sounds like we had the same problem and found the same solution.


All times are GMT -5. The time now is 09:03 PM.