I have a file server that currently has 2.5 million files on it. We add about 1,000 per day and notations can be made on any number of other documents. The server is accessed by mostly Windows clients over samba.

The files are on a RAID 10 array. From there I copy them to a stand by computer in the server room and also to a server in an off site facility.

The problem I have is that currently the process of running the find command to find any of the files that have changed in the past day takes several hours and that does not include the time to then copy the files to the other servers.

I am currently considering dropping the use of samba and making my own socket based solution so I can keep track of the files that are written to throughout the day and instead of backing up to the stand by and offsite servers overnight, back them up as they are modified. But before I go to the effort of doing that, I wanted to see if anyone knew of something I might be missing that would be simpler to implement.

Thanks for any input.

Hello,

I've had very good results with Unison File Synchronizer. It's worth checking it out for what concerns your needs I believe. It'll work a lot faster and better then using a find command. Another advantage is that you wouldn't have to run it once a day, you can keep your copies synchronized automatically (excluding conflicting files) several times a day, in an hour, and so on. Only the first run will take some time in your setup (millions of files) but after that it'll run fast without consuming lots of resources and / or bandwith.

Kind regards,

Eric