LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 07-20-2015, 01:47 PM   #1
mgok
LQ Newbie
 
Registered: Jul 2015
Posts: 1

Rep: Reputation: Disabled
Running a java program on multiple computing nodes at once


I am trying to test if reading a large text file (15GB) is faster on one computer or on multiple. I assume that multiple will be faster, but the lab I am using has been having issues and this test will help determine what is wrong.

I am new to using linux and using ssh to access other computers. What I have so far is two java programs: Time and Read. Time just returns the current date in milliseconds and Read goes through the large text file line by line using scanner. There are 15 computers available for me to use in the lab, and my question is how to run the Read java class at the same time on all 15 computers (hopefully increasing performance speed)?

From my own research using & lets you run multiple jobs at once, but whenever I do a command like: "ssh computer1 java Read &" the terminal says that the main of the program cannot be found. And when I try something like: "ssh computer1 cd /directoryOfFile && java Read &", the job says it is done immediately, which must mean it is not running the program since it takes some time to go through a 15GB file. Any tips?

Thank you!!
 
Old 07-20-2015, 05:00 PM   #2
smallpond
Senior Member
 
Registered: Feb 2011
Location: Massachusetts, USA
Distribution: Fedora
Posts: 3,966

Rep: Reputation: 1210Reputation: 1210Reputation: 1210Reputation: 1210Reputation: 1210Reputation: 1210Reputation: 1210Reputation: 1210Reputation: 1210
Do you have multiple copies of this text file on different disks?
 
Old 07-21-2015, 08:20 AM   #3
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
Quote:
Originally Posted by mgok View Post
I am trying to test if reading a large text file (15GB) is faster on one computer or on multiple. I assume that multiple will be faster, but the lab I am using has been having issues and this test will help determine what is wrong.

I am new to using linux and using ssh to access other computers. What I have so far is two java programs: Time and Read. Time just returns the current date in milliseconds and Read goes through the large text file line by line using scanner. There are 15 computers available for me to use in the lab, and my question is how to run the Read java class at the same time on all 15 computers (hopefully increasing performance speed)?

From my own research using & lets you run multiple jobs at once, but whenever I do a command like: "ssh computer1 java Read &" the terminal says that the main of the program cannot be found. And when I try something like: "ssh computer1 cd /directoryOfFile && java Read &", the job says it is done immediately, which must mean it is not running the program since it takes some time to go through a 15GB file. Any tips?

Thank you!!
Well, putting the run in the background DOES cause it to terminate immediately. 1. the ssh connection is terminated. 2. processes attached to the terminal (stdin/out/error) get terminated automatically on logout. You might try "ssh computer1 'cd /directoryOfFile && nohup java Read &' " might do the job. The "nohup" detaches so that automatic termination doesn't occur - it also reconnects stdout and stderr of the process to the log file nohup generates, and stdin to /dev/null.

As a side note, the speed depends on what your file server is. If it is NFS, it will run faster... as long as the file buffers in memory are the ones called for.

If it is not, then the entire thing slows down as buffers have to be reloaded, the NFS server could run out of service daemons, your network slows down...

It really depends on multiple things - network topology, and then the nature of the file server(s). For clusters, gluster tends to do a much better job than NFS (it can use more memory for cache...), and using multiple gluster servers spreads the network load out.

Last edited by jpollard; 07-21-2015 at 08:24 AM.
 
Old 07-21-2015, 09:11 AM   #4
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Debian, Arch
Posts: 3,696

Rep: Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025
Quote:
Originally Posted by mgok View Post
whenever I do a command like: "ssh computer1 java Read &" the terminal says that the main of the program cannot be found.
Please post the exact error message. That either means it can't find the Read.class file or it found only some of the class files. Is it installed on "computer1" as well?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
adding disks to computing nodes in a diskless cluster mahmoodn Linux - Server 3 12-31-2012 04:52 PM
high performance computing cluster - child nodes speed test m2azer Linux - Networking 1 06-23-2009 03:56 AM
running java program through cron manu82 Linux - Software 2 02-28-2006 10:35 PM
Running a program with multiple sessions Poetics Linux - Newbie 5 12-29-2003 05:57 AM
Running a Java Program from XWindows kilobravo Linux - General 4 01-16-2003 02:36 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 09:45 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration