hi,

i have the following scenario:

<start>
one live fileserver, wich serves millions of small jpg files (photos) with lighty for a big website.
like 400GB of .jpg files in different folders.

and one backup server with a 500gb hdd in the same network with a mounted nfs share from the live server.

i have to copy the whole content from live to backup, without the possibility to take the live server offline.
</start>

When i start rsync on the backup, it takes ages til it starts to copy, and the cpu load is 100%, site will go offline.

When is start cp /live /backup its the same - offline.

the best would be to copy it immediately but only in like 20 files parts.

is that possible? or any other suggestions?


best regards, and thanks!

Sascha

Perhaps try running your copy/backup process with an elevated nice level. For more info:

man nice


--- rod.

i would certainly go with that suggestion...i do this myself what i run is:-
nice -n 20 rsync .....

ps- priorities are -19 highest priority tooo ---> 20 lowest priority. You must be root user to increase (higer priority) but any user can lower priority for a command.

Regards

hi, thanks a lot for the answers.

i allready gave nice a try, but the cpu does still raise up to 100%...
well, the server always has lot of work to do...

i think it would be the best to copy parts of the files...but how can i do that?

regards

Just a note, there isn't really anything wrong with a high cpu load. If the nice value of a process is 20 and it is taking up 100% of the cpu then that just means that nothing else important is needing the processor. Instead of worrying about the cpu load with the copy at nice=20 what is the actual response like from the system?

HTH

Forrest

p.s. What is putting the files on the server? Can it also put them in the correct backup location?

The bottleneck is probably not the CPU. It is more likely the amount of disk I/O or the disk cache being flooded by rsynch or cp. You might be able to reduce the amount of I/O by only copying files that have changed. Use the same /backup directory or directories over and over again. Then use rsynch -u to only copy the files that have changed. Similarly if you use:

cp -u /live /backup

then cp will only copy the files that have changed since the last backup.

--------------------------
Steve Stites

This is more of a question really, but would it be better to tar up the files before you copy it across the network. It seems to me that would save a lot of requests from cp through nfs, which might be the bottleneck. As I said, I'm not sure. I suppose you'd also need a way to create the backup archive (maybe a cron job on the server?).
Comments appreciated.

hi, thanks for all the answers.

i could give tar a try, but i dont have enough space left and i think, it will make the server unavailable till its finished?

Jailbait: you are right, the I/O wait keep the server down, not the CPU load!

Forrest: Users do that (so apache with php), its a big community site, where ppl can upload images - and they do it a lot!


I need a "backup basis" first, with the up to date state.
then it will be not a problem to do a copy -u /live /backup since it will only take a few minutes.

how about a for or a while loop with a file limit? is that possible?

regards

How about changing the php code to write the file to both locations but only removing it from the web server, not the backup location?

HTH

Forrest

yeah that would be a solution if i allready have the backup up to date, but i need to copy 400GB NOW.

S.

You could try (I'm going off of the locations you had in your first post, fix as needed):
HTH

Forrest

going to give it a try! thanks a lot

got the solution.

nice, ionice, rsync