Hello,

i am looking for an easy and efficient way to clone data in a cluster,

i am trying to do a chain of ssh copies:
the 1st node begins to send data to the 2ndnode,
and the 2nd node while is receiving data copy it to hard disk
and also send it to the next node, and so on

something like this:

tar cf - dir_tobecopied | ssh node2 "common_script"

where common_scrip is a script that contains:

mkfifo temp
(tar xf temp ) &
tee temp | ssh $next_node "common_script"

i think this way i should be able to use as much as possible the switch
backbone and reduce the time needed for it,

for little files (less than 50MB) it worked fine :-) but for a bigger
ammount of data i found this error :

tee: read error: Resource temporarily unavailable
... and it fails

(i frecuently need to clone 10 GB of data in around 100 nodes
with gigabit ethernet)
any help please ?

other posibility is to use multicast but i could not found any good
working tool for it,

regards,

Rafa

It seems to me that bittorrent might be the perfect solution here:

http://www.bittorrent.com/

thanks for you answer tredegar, but i am afraid bittorrent is not the best solution to my problem,
i found it quite complex for what i really need,

besides i forgot to comment an important detail: the machines are distributed between several switches
with only a 1 gigabit ethernet conexion, if i would use bittorrent ie. all the machines from the second switch
would try to download data from the first switch and it would create a big bottleneck, using a chain of
simultaneous copy commands would elude that problem,
(the switches i am using have 48 gigabit ports and 2x48gigabit backbone, but conexion between them
is just 1 gigabit)

regards, Rafa