LinuxQuestions.org
Visit Jeremy's Blog.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 10-22-2014, 07:53 AM   #1
ricky-wilson
LQ Newbie
 
Registered: Oct 2014
Posts: 2

Rep: Reputation: Disabled
Queue of Commands for distributed precessing


I have a php web page running on a low powered web server (home use only) that writes FFMPEG commands to a file for later processing by a more powerfull machine. It possible to write a shell script, to run on the more powerfull machine, that will read the first line of the queue (all relivant files are available on an samba share), remove that line from the file, execute it and repeat until the file is empty. I could then run this on multiple computers to encode at the same time (I have potentially a couple of thousand files to encode).All machines are running on Current Debian.

Extract forom Queue File(with file paths removed)

Path_to_FFMPEG -threads 2 -i Path_to_uploaded_CDG -i Path_To_Uploaded_MP3 -f webm -b:v -acodec libvorbis -vcodec libvpx -preset slower -qp 0 -sws_flags print_info+neighbor -s 640x480 -r 30 -y Target_File_For_Output_webm
/home/richard/www/upload/ffmpeg -threads 2 -i Path_to_uploaded_CDG -i Path_to_uploaded_MP3 -f webm -b:v -acodec libvorbis -vcodec libvpx -preset slower -qp 0 -sws_flags print_info+neighbor -s 640x480 -r 30 -y Target_File_For_Output_webm
 
Old 10-22-2014, 09:45 PM   #2
btmiller
Senior Member
 
Registered: May 2004
Location: In the DC 'burbs
Distribution: Arch, Scientific Linux, Debian, Ubuntu
Posts: 4,290

Rep: Reputation: 378Reputation: 378Reputation: 378Reputation: 378
It's certainly possible, but it might require some custom processing. Also, you'll have to be careful that both systems don't try to update the file at once. I'm not sure how well CIFS handles file locking, so that might be an issue.

You might want to look at free distributed queuing systems such as TORQUE, Open Grid Engine, or SLURM. They're not quite designed for your use case, but they could be used. I think most big render farms use DrQueue, so that one might be worth checking out too, but I have no personal experience with it.
 
Old 10-23-2014, 07:34 AM   #3
ricky-wilson
LQ Newbie
 
Registered: Oct 2014
Posts: 2

Original Poster
Rep: Reputation: Disabled
Thanks btmiller I'll check out those queuing systems.
To avoid the possible issue of accessing a file by 2 machines at once I could flip the whole thing around a bit. If the server pinged a list if possible IP's and from that created a list of active machines, it could iterate through them and using ssh, first test the client machines 5 min average CPU load using upstart if it is less than say 80% it would send the FFMPEG command and then move onto the next ip address, if all client machines are over 80% load wait 60 sec and repeat.
It would mean setting up preshared keys which I've never done but there are countless tutorials on line.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
difference between ready queue and run queue soumya_r Linux - Kernel 2 01-11-2011 09:19 PM
Any chance to get length of run queue and swap queue? emJ4y Programming 1 11-03-2010 02:13 PM
How to untar my tarred mail queue folder back to the sendmail queue directory again? Md.Abul Quashem Linux - Server 6 02-16-2010 08:32 AM
Is it possible for me to write in front of queue in case of message queue? hemanthv414 Linux - Newbie 1 11-17-2008 04:40 PM
Deleted qmail queue /var/qmail/queue tris Linux - Software 0 10-24-2003 11:52 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 03:53 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration