I'm having a rough time at work at the minute pushing a tar backup to tape.
I've tried to rack my brains with what the frell is wrong with my server.
The nitty gritty is this:
HP DL380 G4
Dual 64-bit Xeon 3.4GHz
SmartArray 6i U320 (& 128MB BBWC i think)
+ 4x 146GB 10K U320 in RAID 5
Adaptec U160 PCI-X SCSI Card (unsure of model)
+ External IBM Ultrium 3 Tape Drive
Ubuntu 9.10 x64
All hardware seems pretty ok, nothing missing from usual places like /proc/scsi
I think the RAID 5 stripe size is the reccommended 64KB.
I'm using tar to dump /filestore to tape (/filestore being the root of the RAID 5 partition)
htparm -t -T /dev/cciss/c0d1p1 (/filestore) yields values in the following ranges:
Cached Reads 950 ~ 980 MB/s
Buffered Reads 30 ~ 35 MB/s
Apologies if buf/cache wrong way round
Basically I'm getting awful performance dumping ~150GB of data to /dev/tape. It takes in the order of about 3hrs 30mins. That tells me about 11MB/s. Now i've read that the tape drive's data rate matching (finally a good DRM!) is good from 80MB/s native down to 30MB/s after that its shoe shine time and I know thats what I'm getting because I can hear spin ups and spin downs a lot (no cockney accents though!)
I ran some tests running the system through mbuffer (many thanks to its author for a good program) and a 1GB memory mapped i/o file. I ran the simulation into /dev/null and put a cap on the stdout from mbuffer at 40MB/s ( using pv -q -L 40M )to stop the bit bucket munching the buffer too quick and it is a reasonable rate I would like to see going to tape. (Half speed isn't too upsetting!)
A block diagram i guess looks like this:
/filestore ---tar---> mbuffer ---> pv [Lim 40MB/s] ---> /dev/null (test)
/\ /dev/tape (real)
I've got mbuffer to cache 800MB before starting to stream out data.
I used the logging feature of Putty (another good tool) to keep a nice record from mbuffer's stats, parsed this in MS Excel and graphed the data (I miss gnuplot).
-- side note: how often does mbuffer produce output - looks like 1/2 sec
For the first 5000 entries, all is good, I'm not sure where I am in the filestructure of /filestore, its full of Adobe Suite files [ai, psd, etc], web source files [php/asp], images [png/jpg] and SVN repos/dumps.
There are a couple of buffer underruns but largely the buffer stays full.
After this point the whole lot goes down hill. I must move into another par of the filestore where things aren't so easy and I start to get more and more buffer underruns. After a bit of this the average graphed input rate drops to 12MB/s and its game over, a bit later it drops again to 9MB/s.
Obviously I can't backup this to tape without some seriously shiney shoes! (And associated tape wear)
I'm not looking neccessarily for someone to tell me a full solution but the way things are looking, its going to be a case of buying another server with a big RAID 0 striped array on it and thrash everything to there, once staged on there I can back up to tape nice and fast.
Now for the questions:
1. Is there something I have missed, blaringly obvious? I'm not the oldest or wisest sysadmin around.
2. Does anybody have or have experience with a similar setup and can give me some idea of what this lump of steel should be able to do. I'm guessing a SmartArray 6i and RAID 5 should be faster than 30MB/s.
3. Is there a better way of backing up than a straight tar? I have heard of dump(?) to dump the whole unmounted filesystem but that doesn't sound like what I need. I'm not really able to unmount the filestore for any time and I often need to pull back a couple of files off tape when they go "missing"
4. Is it going to be a case of forking out for an external SCSI drive array and running a fast RAID 0 striped staging drive on it? (And get ready for my boss' expression!)
5. Can anybody shed any other light on this?
I can try to get you any logs / outputs you need, if you need them.
Thanks all for your help in advance and for keeping a great community!