scp and filename manipulation

karnaf · 08-23-2011, 01:52 AM

Hi all,

I'm writing a bash script which collects files from several servers into a single folder (e.g. logs). In order to avoid files with the same name but from different servers overriding each other, I want to prepend the server's name before them, e.g.

Code:

scp user@server1:/mylogs/filename.log /targetlogs/server1-filename.log

however, my source filename isn't known in advance, and there may be more than one. i.e. I'm using

Code:

for server in "${SERVERS}" do
   eval "scp user@$server:/mylogs/* /targetlogs/$server-<HERE IS WHERE YOUR HELP IS NEEDED>"
done

How can I get the source filename to the target location?

Currently, I'm thinking using a workaround (copy to a tmp folder, rename it there locally, and then move it to its final destination

Thanks!
karnaf

colucix · 08-23-2011, 04:03 AM

The scp command does not offer any option to add a prefix to the filename, hence you need some scripting. A simple loop fed by the result of the ls command on the remote machine should be enough:

Code:

for server in "${SERVERS}"
do
  while read file
  do
    scp -p user@$server:/mylogs/\'"$file"\' ./$server-"$file"
  done < <(ssh user@$server "ls /mylogs")
done

Note the escaped single quotes in the scp command: they are useful if the file name contains blank spaces: they don't prevent the substitution of $file in the local server because they are treated literally and not a a special character. On the other hand they prevent the word splitting onto the remote server. Hope this helps.

karnaf · 08-23-2011, 05:41 AM

Funny, I ended up using a similar solution

Code:

for server in "${SERVERS[@]}" 
do
        files=`ssh $SOURCE_USER@$server "ls $SOURCE_PATH/*.$SOURCE_SUFFIX" 2>&1`
        
        for file in $files
        do
                cmd="scp $SOURCE_USER@$server:$SOURCE_PATH/$file $TARGET_PATH/$server-$file"
                eval "$cmd"
        done
done

cmd is kept in a variable for debugging purposes

Thanks for the help!
karnaf