LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 06-06-2013, 08:57 AM   #16
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,520

Rep: Reputation: 2106Reputation: 2106Reputation: 2106Reputation: 2106Reputation: 2106Reputation: 2106Reputation: 2106Reputation: 2106Reputation: 2106Reputation: 2106Reputation: 2106

There are two main ways to run things in the background. You seem to be hitting bits and pieces of each of them, but never doing either correctly.

1) Run it in the background with nohup
Code:
nohup command > logfile 2>&1 &
You've tried this, but it looks like you keep missing the trailing "&". You don't seem to be running exactly what is being suggested here, which could be the problem. Run EXACTLY this, do not leave anything off:
Code:
nohup mpirun 4 MYPROG 6 0 > MYPROG.log 2>&1 &

or

2) Put the process in its own screen session
Code:
screen -S MYPROG_screen bash
That launches a new shell inside the screen (and names the screen "MYPROG_screen", you can name it whatever you want), then you can run your program in the FOREGROUND (you seem to keep trying to run it in the background?):
Code:
mpirun 4 MYPROG 6 0
Your program will start and do its thing, then whenever you want to detach from the screen just run:
Code:
Ctrl-a d
When you want to reattach, run:
Code:
screen -r MYPROG_screen
I use both of these on a daily basis to do things very similar to what you're doing (multi-processor codes run in the background with mpirun, etc). You just have your syntax a little off.

Last edited by suicidaleggroll; 06-06-2013 at 09:01 AM.
 
1 members found this post helpful.
Old 06-06-2013, 10:05 AM   #17
Jeff9
Member
 
Registered: Jun 2013
Posts: 36

Original Poster
Rep: Reputation: Disabled
Thanks, eggroll. But I still can't get the Ctrl-a d to work. When I try it the terminal just displays "^Ad", but doesn't detach the screen. I'm using "ssh" through Cygwin. Any idea why Ctrl-a d isn't working?

Code:
()> screen -S testrun bash
(testrun)> cd $HOME/sundials/instdir/examples/cvode/parallel
(testrun)> mpirun -machinefile $HOME/utils/Host_file -np 4 MYPROG 6 0
PID 28875 executing on node8.
PID 4254 executing on node7.
PID 2521 executing on node6.
PID 1961 executing on node9.

 Simulating 64x64 oscillators for 9999999 timesteps

 Number of PEs =   4

t = 00000. Overwriting files: /raid/fpuData/N=064x064_Beta=0.00_t=0000000_T.csv,        /raid/fpuData/N=064x064_Beta=0.00_t=0000000_U.csv
^A
See the "^Ad", that's me trying to type Ctrl-a d. But nothing happens.

Note that if I am not running a program (that is, if I'm at the command prompt), then Ctrl-a d terminates the window.

Any help? Thanks.

Last edited by Jeff9; 06-06-2013 at 10:25 AM.
 
Old 06-06-2013, 10:26 AM   #18
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,520

Rep: Reputation: 2106Reputation: 2106Reputation: 2106Reputation: 2106Reputation: 2106Reputation: 2106Reputation: 2106Reputation: 2106Reputation: 2106Reputation: 2106Reputation: 2106
Odd, I run screen on Linux boxes when SSHd in from Cygwin all the time and have never run into an issue with ctrl-a d.

Either way, once the process is running in the screen session, it's safe. You don't HAVE to disconnect using the screen key commands, you can just close the cygwin terminal if you want.
 
1 members found this post helpful.
Old 06-06-2013, 11:00 AM   #19
Jeff9
Member
 
Registered: Jun 2013
Posts: 36

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by suicidaleggroll View Post
Odd, I run screen on Linux boxes when SSHd in from Cygwin all the time and have never run into an issue with ctrl-a d.
Edit:I just tried an application called PuTTy, Ctrl-a d didn't work for me in that application either. IOW, Cygwin is not the problem.

Quote:
Originally Posted by suicidaleggroll View Post
Either way, once the process is running in the screen session, it's safe. You don't HAVE to disconnect using the screen key commands, you can just close the cygwin terminal if you want.
I verified that this is true.

Code:
()> screen -S testjustclosescreen
(testjustclosescreen)> mpirun 4 MYPROG 6 0
PID 2544 executing on node9.
PID 29458 executing on node8.
PID 4844 executing on node7.
PID 3108 executing on node6.

 Simulating 64x64 oscillators for 9999999 timesteps

 Number of PEs =   4

t = 00000. Overwriting files: /raid/fpuData/N=064x064_Beta=0.00_t=0000000_T.csv,        /raid/fpuData/N=064x064_Beta=0.00_t=0000000_U.csv
(close window)
Wait twenty minutes, open new Cygwin, ssh into the cluster.
Code:
()> screen -li
There are screens on:
        20935.testjustclosescreen       (Detached)
1 Sockets in /var/run/screen/S-<myusername>.
()> screen -r testjustclosescreen
(testjustclosescreen)> mpirun 4 MYPROG 6 0
PID 2544 executing on node9.
PID 29458 executing on node8.
PID 4844 executing on node7.
PID 3108 executing on node6.

 Simulating 64x64 oscillators for 9999999 timesteps

 Number of PEs =   4

t = 00000. Overwriting files: /raid/fpuData/N=064x064_Beta=0.00_t=0000000_T.csv,        /raid/fpuData/N=064x064_Beta=0.00_t=0000000_U.csv
t = 10000. Overwriting files: /raid/fpuData/N=064x064_Beta=0.00_t=0010000_T.csv,        /raid/fpuData/N=064x064_Beta=0.00_t=0010000_U.csv
It picked up right where it left off, and all of the output is there. It may not be as elegant as we would like, but it works for me.

Last edited by Jeff9; 06-06-2013 at 11:08 AM.
 
Old 06-06-2013, 06:22 PM   #20
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,704

Rep: Reputation: 1270Reputation: 1270Reputation: 1270Reputation: 1270Reputation: 1270Reputation: 1270Reputation: 1270Reputation: 1270Reputation: 1270
Cluster batch systems have a data collection part that allows the logs to be collected for the entire job. So errors from mpi services on remote nodes can get collected as well as the processes errors that are directly spawned.

I'm not familiar with how all of them have worked, but only PBS (and I only worked with it for security services, not the distributed log/submission/management part)- which can handle the migration of processes from node to node, restarting jobs from checkpoints, and in general, handling things that a process doesn't do well.

One thing they do allow is for management - it allows a cluster to be partitioned to allow multiple jobs to be run simultaneously, and allow for exclusive access to nodes, or shared access. It also allows the user to submit jobs based on priority. Granted, most of the facilities are more useful as the cluster gets larger, but even small clusters can use some of the features even if only one queue is defined, and only one job at a time allowed. It gives the user independence of having to monitor the job manually... and can sometimes be able to restart a job automatically due to a system wide failure (such as a network failure... once the network is restored the job may be resumed, or restarted).
 
1 members found this post helpful.
Old 06-07-2013, 09:13 AM   #21
Jeff9
Member
 
Registered: Jun 2013
Posts: 36

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by suicidaleggroll View Post
Odd, I run screen on Linux boxes when SSHd in from Cygwin all the time and have never run into an issue with ctrl-a d.

Either way, once the process is running in the screen session, it's safe. You don't HAVE to disconnect using the screen key commands, you can just close the cygwin terminal if you want.
Note also that you can sign on to another terminal window and issue screen -d.
 
Old 06-07-2013, 05:55 PM   #22
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,704

Rep: Reputation: 1270Reputation: 1270Reputation: 1270Reputation: 1270Reputation: 1270Reputation: 1270Reputation: 1270Reputation: 1270Reputation: 1270
Note the "...It picked up right where it left off...".

During the time it was disconnected, I believe the processes were also blocked (can't output).
 
Old 06-07-2013, 06:05 PM   #23
Jeff9
Member
 
Registered: Jun 2013
Posts: 36

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by jpollard View Post
Note the "...It picked up right where it left off...".

During the time it was disconnected, I believe the processes were also blocked (can't output).
No, they kept processing. In fact, I'm processing a 9999999 step job right now. I should have phrased my reply more carefully.

The jobs kept processing while I was detached (screens detach instead of disconnect) and the display (stdoutput and sterror) picked up where they left off. screen keeps running as if it's in the foreground (or at least that's how it seems to me). In fact, if you add "-L" option, screen will log everything to screenlog.0 (although I just now noticed that there is also a screnlog.0~, which I don't know what that is).

Last edited by Jeff9; 06-07-2013 at 06:07 PM.
 
Old 06-07-2013, 11:46 PM   #24
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.9, Centos 7.3
Posts: 17,395

Rep: Reputation: 2395Reputation: 2395Reputation: 2395Reputation: 2395Reputation: 2395Reputation: 2395Reputation: 2395Reputation: 2395Reputation: 2395Reputation: 2395Reputation: 2395
Re 'screenlog.0~'; files ending in '~' are usually created as temp/point-in-time backup/recovery files, in case the 'primary' is lost.
HTH
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Running program in the background HarryBoy Programming 2 06-12-2008 11:42 AM
Running C Program in Background help ibshar Linux - Newbie 7 10-04-2007 02:41 PM
Running a program as a background daemon? Vor Kragresh Slackware 8 04-20-2007 10:48 AM
running a program in background of X using & Dachy Linux - General 1 09-13-2005 11:38 AM
Running a program in the background - SSH ziggo0 Linux - Newbie 4 03-05-2005 01:30 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 05:53 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration