multiple ubuntu questions (automation related)

mrgreaper · 04-25-2010, 04:32 PM

ok i have used ubuntu for a while now but i still consider myself a newb.

ok first some background and possibly relevent information

the server pc is a desktop version of ubuntu (9 i think) and it starts itself at 2100 and is shut down nightly before bed (0300ish)

the target netbook (ill explain that later) a n110 runs ubuntu remix (9.10 i think)

I use sabnzbd to get my tv episodes this is as an alternative to my dvd recorder (my netbook has no dvd drive) or my virgin media hd box (can not do anything with the stuff on there) no laws are broken, i have a license i have a full tv subscription to virgin media.

ok heres what happens at the moment,
episode butler tags them,
sabnzbd downloads them into a episode folder inside a season folder inside a show folder
a post script duplicates the show into a holding folder if its an avi

along comes me i connect my netbook to the network and copy the contents of the holding folder to its hard drive, go to work and watch them

now this is a problem if the download fails, i have no idea unless i check the folder or as today at work ,think..hmm wheres doctor who? its also messy as i need two copys of the file (even if it is at times for no more then an hour, and creates additional hd access

so heres what i want to do but have no idea on the method

a script that is run every 30 minutes

1)checks to see if a shared folder on the target netbook is connected
if yes continues
if no exits script till next time

2) scans my tv folder and makes a list of all *.avi files in said folder (lets call that current.txt)

3) compares this to old.txt (created later so manualy created for the first run) creating tobecopied.txt which is all the files that are in current.txt that are not in old.txt

4)copys all the files from tobecopied.txt to the netbooks shared folder

5)deletes old.txt and tobecopied.txt

6)renames current.txt to old.txt

7)scans the folder for the words failed or FAILED or ipod or mp4 or mkv or wmv or password and adds them to a errors.txt (with full path name) in my home folder (or even better yet emails me the list ..i know not possible but wouldnt that be good?)

8) ends the script till another 30 minutes.

any help suggestions would be greatfully recieved, im just trying to make it all automatic and neat

EricTRA · 04-26-2010, 12:18 AM

Hi,

The easiest and quickest way is to post what you already have in regards to scripts and tell us what parts are not working. You cannot expect us to do everything for you. We're good at solving problems, answering questions and so on. LQ, as far as I know doesn't provide ready made solutions or custom work. If you don't have any experience in scripting, then read something like the guides mentioned below and take it from there.

Good starting points are these:
Bash guide for beginners
Advanced Bash scripting guide

Kind regards,

Eric

mrgreaper · 04-26-2010, 10:23 AM

will read the guides,

i just wish linux was as easy as easy amos on the amiga lol

if such and such goto do

its kinda not knowing where to start and i had to sort some other stuff last night so didnt have time to try stuff

EricTRA · 04-26-2010, 10:29 AM

Hi,

Basically, if you know your way around the console, running commands to list, move, copy files then you have the basics covered to start with shell scripting, since a shell script is nothing more than a list of shell commands grouped together in order to automate things like the stuff you want. If you know how to do it manually then you can start putting everything together in a script. If you're a GUI guy, then I suggest you start with the basic commands first.

Kind regards,

Eric

mrgreaper · 04-26-2010, 04:52 PM

ok im stuck i dont even know what to google, no idea how to do it. im still hunting but i thought i would post the script as is.

Code:

#!/bin/sh

#this section will check to see if the netbook is connected to the network and if it is then it will continue if not it will exit
#yeah i have no idea how to do it ...i know the exit command..its exit
# check smb://ickle/inbound if there goto next step if not then exit

###

#here we set the local folder up
tv=/home/mrgreaper2004/Tv/

#now we create the current tv show list (all the avi`s in the folder sadly it doesnt give me a full path to each one which im sure would be 10 times more helpfull but this line is a butchered example that i cant figure out how it works
tv2="$(find $tv -type f -name "*.avi" -exec basename "{}"   \;)"

#i then needed to load all the names from the old episode list (one created at the end of the script) after 2 hours of googling i finaly hit on SED which will aparently do just that...it says i dont have permission "./test.sh: 7: /home/mrgreaper2004/test2.txt: Permission denied" grrrrrr

old=sed /home/mrgreaper2004/test2.txt

#right next we compare the new list with the old list and place all the unique programes into another varriable..again it would be useful to have the full paths

new="$(echo -e "$tv$old"|sort -d|uniq -u)"
#echo $old
#next to copy the filed 
cp new smb://ickle/inbound

#for testing purposes saves the current list
echo "$tv2" > /home/mrgreaper2004/test.txt

#now to create the list of old episodes for use with the next time the script is run
echo "$tv2" > /home/mrgreaper2004/test2.txt

exit

the new error when i run this is
./test.sh: 17: /home/mrgreaper2004/test2.txt: Permission denied
cp: cannot stat `new': No such file or directory

ummm help?

ok i chmod 777 test2.txt and i get a new error

/home/mrgreaper2004/test2.txt: 1: test.avi: not found
/home/mrgreaper2004/test2.txt: 2: test3.avi: not found
/home/mrgreaper2004/test2.txt: 3: test2.avi: not found
cp: cannot stat `new': No such file or directory

mrgreaper · 04-26-2010, 09:02 PM

found out to check a folders existance on my test net book that part of the script works fine till i try putting a smb:// folder then even though i know it exists, it tells me it does not

Code:

#!/bin/sh

#this section will check to see if the netbook is connected to the network and if it is then it will continue if not it will exit
#yeah i have no idea how to do it ...i know the exit command..its exit
# check smb://ickle/inbound if there goto next step if not then exit
folder=/home/mrgreaper2004/testfold
 
if [ -d $folder ];
then
   echo "script says $folder exists success"
else
   echo "$folder cant be found..fail"
fi
###

#here we set the local folder up
tv=/home/mrgreaper2004/Tv/

#now we create the current tv show list (all the avi`s in the folder sadly it doesnt give me a full path to each one which im sure would be 10 times more helpfull but this line is a butchered example that i cant figure out how it works
tv2="$(find $tv -type f -name "*.avi" -exec basename "{}"   \;)"

#i then needed to load all the names from the old episode list (one created at the end of the script) after 2 hours of googling i finaly hit on SED which will aparently do just that...it says i dont have permission "./test.sh: 7: /home/mrgreaper2004/test2.txt: Permission denied" grrrrrr

old=sed /home/mrgreaper2004/test2.txt

#right next we compare the new list with the old list and place all the unique programes into another varriable..again it would be useful to have the full paths

new="$(echo -e "$tv$old"|sort -d|uniq -u)"
#echo $old
#next to copy the filed 
cp new smb://ickle/inbound

#for testing purposes saves the current list
echo "$tv2" > /home/mrgreaper2004/test.txt

#now to create the list of old episodes for use with the next time the script is run
echo "$tv2" > /home/mrgreaper2004/test2.txt

exit

i need sleep will work more on it when awake

mrgreaper · 04-27-2010, 07:22 AM

well i have spent 3 hours working on it again today and i cant find any help i just have no idea what im doing

im now looking at alternatives,

an rss feed created by the server pc of the folder date ordered and linking directly to the files so i can use some sort of automated rss feed grabber to grab the latest files

-or-

something called svn (just found out about it) will keep my folder synced with an external folder ((advantage keeps the same folder structer, disadvantage might copy files i have already watched onto my netbook and files i dont want)

i really could use some guidence

EricTRA · 04-27-2010, 07:35 AM

Hello,

If you are looking for something to synchronize two folders on different machines and that's highly configurable, have a look at Unison File Synchronizer. It's cross-platform and highly configurable. I use it in a production environment and it really does the job. Furthermore it's even not so hard to configure and comes with a pretty good manual. Most important of course is that is Open Source and comes completely free.

Kind regards,

Eric

mrgreaper · 04-27-2010, 08:32 AM

thank you.

i wish i could get the script to work as that would be the most ideal solution but failing that your linked application will do the trick.

on the script front i discovered that "find" is a command and i have been playing around with

Quote:

tv2="$(find $tv -type f -mmin 1200 -name "*.avi" )"
echo $tv2

to get it to list all files modified within the last 1200 minutes and tha would give me a list of new files to comparer against the list of old files

but then sed isnt loading the txt file into a varriable how the manual hints it does, i dont know if my compare line will work
and the part of the script that checks for a folders existence only works if looking for local folders not remote folders (i even fort about creating a folder that symlinks to the smb folder at the start of the script and then having it unsymlink it at the end of the script so i would have a local folder the test for existence part could see unless symlink errored, but the manual for that is written for brain surgeons not mere mortals)

why does it all have to be so newb unfriendly!!!arghhhhhhhhhhhhhhhhhhhhhhhhhh

EricTRA · 04-27-2010, 08:39 AM

Hi,

For comparison between two files you could use comm, if both files are sorted. Normally as far as I know if you want to compare then you do that line by line, using a loop structure in your script. Maybe I'm mistaking but I don't think you can load an entire file in a variable and then compare it to another one. As said look at comm (man comm) to compare two sorted files.

As for the 'newb unfriendly' statement, hey that comes with the territory of having FULL access to EVERYTHING on your system, instead of being controlled by M$. Of course with that level of control comes a 'steep' learning curve. Believe me, it's worth it.

Kind regards,

Eric

mrgreaper · 04-27-2010, 09:03 AM

true, it just frustrates me i can see what i want it to do is possible, but cant figure it out

i spent an entire shift at work then 3 hours at home yeasterday and since i got up 5 hours ago today, looking for the solution and im no where close to finding it

all i want is

check the smb share is on the network and exit if its not (that part of the script only works on local files)

make a varriable with a list of avi files in the folder

load a prevously saved list of avi`s that where in the folder last time the script was run

compare variable a and variable b making a list of all avis that where in the first list but werent in the old list

copy the files listed in above list to the samba share

seems so simple and would save me a lot of time

EricTRA · 04-27-2010, 09:08 AM

Hi,

One thing comes to mind when re-reading your first post. You say that both your server and netbook are Ubuntu, yet you use SMB to connect to the directory on the server? SMB is part of the Samba suite which is used to interconnect Linux and Windows filesystems.

Kind regards,

Eric

grail · 04-27-2010, 09:32 AM

Hi, my first time looking at this so hopefully you will allow me some lee way.
I would like to address the points you listed in post #11, which by the way is great way to start and I would have suggested
using this approach earlier and only progressing to the next item you want after conquering each at the command line and adding to your script.

Anyhoo:

1.

Quote:

check the smb share is on the network and exit if its not (that part of the script only works on local files)

Would have to agree with Eric, why go to a Windows tool to connect to linux machines??

2.

Quote:

make a varriable with a list of avi files in the folder

As I see this question you actually have two choices (well probably loads, but two that stand out)
a. Use an array - so like you have done so far but place it in an array like so:

Code:

array_name=($(find $tv -type f -mmin 1200 -name "*.avi"))

b. Simply pipe output of find into a loop, I would go for a while loop (this one is probably my preferred choice)

Code:

while read line
         do
             <do your stuff here>
         done< <(find $tv -type f -mmin 1200 -name "*.avi")

3.

Quote:

load a prevously saved list of avi`s that where in the folder last time the script was run

Now again here there are many options:
a. Create an inner loop to the one above with the same format but feed the file instead of the find into the "done" at the end
b. Now there are many tools but the simplest is probably to grep the above "line" from the file with the previous data and do an "if" on the
result. true ignore, false put it in ... or something like that

4.

Quote:

compare variable a and variable b making a list of all avis that where in the first list but werent in the old list

See previous step

5.

Quote:

copy the files listed in above list to the samba share

Again not sure about the samba part, but if still required we can go there.
This would also be part of the if in step 3 above to now cp the required file where it needs to go.

Hopefully I haven't confused you too much. Ask away if I have lost you anywhere?

I recommend the following site: http://tldp.org/LDP/abs/html/ (don't let the name fool you its good for all levels of user)
Here I would look up : Arrays and Here Documents ... and if required Loops & Branches

Cheers
Grail

mrgreaper · 04-27-2010, 10:13 AM

im in the middle of getting ready for work but checked this and above sounds intriging and ill read in more detail after my first patrol (though tuesday night is a busy night so it may take time (dont you hate when work interfers with stuff))

its all in samba shares as i thought that was the linux version of networking lol (renember linux newb)
if there is a better way of sharing a folder of my netbook with my server pc, believe me im all ears

(well eyes)
in the olden days i used windows laptop (my mates still do) for work but i find ubuntu remix to be much smoother on my little netbook then windows.

grail · 04-27-2010, 10:28 AM

I believe NFS is the way to go here and it will then just appear as part of your system, once mounted of course

here is a link from a quick search which seems to lay out what you will need:

http://mybeni.rootzilla.de/mybeNi/20..._ubuntu_linux/

Let me know how you get on