LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices



Reply
 
Search this Thread
Old 01-26-2007, 08:14 AM   #1
frankie_DJ
Member
 
Registered: Sep 2004
Location: NorCal
Distribution: slackware 10.1 comfy, Solaris10 learning
Posts: 232

Rep: Reputation: 30
bash script problems: scp/ssh from the node of a cluster to the other server


Hi everyone,

I am running a series of simulations on a cluster node. The output of the simulations are some pretty large data files, so I would like to transfer them to the storage server as soon as they are made. I would like to do this in a bash script (there are 68 simulations being run, two at a time, each lasting 2-3 hours so I do need to automate).

The problem is, I can't ssh or scp from the node to the storage server.

(Yes, I have set up and ssh identity file in my home dir on the cluster head and the authorization_key in the home dir on the storage server, so that I am not being asked for password when I scp through the script).

Since node doesn't communicate with the outside world, I am assuming I need to go back to the head and scp the file, then go back to the node to do another simulation. All this in a script. Can anyone help me with this? I tried

Code:
logout #back to the head
scp files storage_server
rsh node10 #back on the node
cd wherever/i/need/to/be/on/the/node
This is put in a loop, so it's done everytime simulation finishes.'logout' doesn't work cause the shell I am in on the node is not a login shell. So I need to use 'exit', but (since exit is a keyword for terminating the loop) that will get me out of the 'for i in ...' loop prematuraly. See what I'm saying? Kinda tricky.

Any help would be appreciated.

Last edited by frankie_DJ; 01-26-2007 at 08:16 AM.
 
Old 01-27-2007, 06:56 AM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,791
Blog Entries: 54

Rep: Reputation: 2979Reputation: 2979Reputation: 2979Reputation: 2979Reputation: 2979Reputation: 2979Reputation: 2979Reputation: 2979Reputation: 2979Reputation: 2979Reputation: 2979
Maybe do the master and servant thing. Separate functionality. You don't need to be on the node all of the time. On the node I would run an independant script that accepts jobs from head and returns their status. On head I would run the "master" script that does job management like accepting a list of jobs from you, scheduling jobs on the node and polling their status and initiating transfers on job finish. This way you don't need to move back and forth which is resource efficient and allows you to loose your sanity in other ways :-]
 
Old 01-27-2007, 07:29 PM   #3
frankie_DJ
Member
 
Registered: Sep 2004
Location: NorCal
Distribution: slackware 10.1 comfy, Solaris10 learning
Posts: 232

Original Poster
Rep: Reputation: 30
unSpawn,

Thanks. That's what I ended up doing. I did lose my sanity first though
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
bash script: ssh to server and reply to any questions cambie Programming 2 09-28-2006 10:44 AM
Need to SSH into cluster node by machine name !? nuro305 Linux - Networking 3 06-23-2006 11:41 PM
Shell Script with ssh & scp ErikJohnson Linux - Software 3 03-11-2004 05:53 PM
Shell Script with ssh & scp ErikJohnson Linux - Networking 1 03-11-2004 05:14 PM
how do I SSH into a cluster node? dogma Linux - Newbie 4 05-15-2003 05:31 PM


All times are GMT -5. The time now is 10:22 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration