LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices



Reply
 
Search this Thread
Old 06-02-2008, 04:06 AM   #1
XeroXer
LQ Newbie
 
Registered: Jun 2008
Location: Västerås, Sweden
Distribution: Arch Linux, Debian, Ubuntu
Posts: 21

Rep: Reputation: 16
Question Reading and writing a file with bash


Hi all,
this is my first post so I would first like to say hi to everyone before asking my question...



I am in the beginning of learning bash scripting and haven't really gotten the hang of reading and writing files.

What I am trying to create is a bash script that by crontab is run every hour, it then checks a text file for links, wget's them all and then marks them as done.

Let's say I have this textfile /home/user/urls :
http://www.example.com/file_1.zip
#http://www.example.com/file_2.zip
http://www.example.com/file_3.zip
#http://www.example.com/file_4.zip
http://www.example.com/file_5.zip

Then my bash script is in /home/user/bash/wget.sh :
exec < /home/user/urls
while read line
do
echo $line
done

That is all I have done so it just prints the line, and I could of course just make it wget $line but I thought I want some more working before I do that.

Now what I was thinking is that it opens the file, loops through every line and on the lines not starting with # it does wget (the line).
After it has downloaded the url it adds a # before that line and continues the loop.

This would make it so if I wanted a file downloaded I would just put the url in this file and the script would within an hour (or any other crontab time) download the file and add a # first on the line so I know it's done.


Any help would be great
Thanks...

Last edited by XeroXer; 06-02-2008 at 05:23 AM.
 
Old 06-02-2008, 04:46 AM   #2
Alien Bob
Slackware Contributor
 
Registered: Sep 2005
Location: Eindhoven, The Netherlands
Distribution: Slackware
Posts: 5,393

Rep: Reputation: Disabled
Something along this perhaps:
Code:
#!/bin/sh
INFILE="/home/user/urls"
for line in $(cat $INFILE | grep -v '^#'); do
  # download the file:
  wget -nv $line
  # mark the line as done by putting '#' in front:
  sed -i -e "s?^$line?#$line?" $INFILE
done
Eric
 
Old 06-02-2008, 05:22 AM   #3
XeroXer
LQ Newbie
 
Registered: Jun 2008
Location: Västerås, Sweden
Distribution: Arch Linux, Debian, Ubuntu
Posts: 21

Original Poster
Rep: Reputation: 16
Thumbs up

Quote:
Originally Posted by Alien Bob View Post
Something along this perhaps:
Code:
#!/bin/sh
INFILE="/home/user/urls"
for line in $(cat $INFILE | grep -v '^#'); do
  # download the file:
  wget -nv $line
  # mark the line as done by putting '#' in front:
  sed -i -e "s?^$line?#$line?" $INFILE
done
Eric
Not that I understood all of that but thank you Alien Bob (Eric).
It works just like I wanted it to and I can now add it to my crontab.
 
Old 06-02-2008, 06:06 AM   #4
Alien Bob
Slackware Contributor
 
Registered: Sep 2005
Location: Eindhoven, The Netherlands
Distribution: Slackware
Posts: 5,393

Rep: Reputation: Disabled
Next on your list should be to understand this bit of shell script completely :-)

Eric
 
Old 06-02-2008, 06:57 AM   #6
XeroXer
LQ Newbie
 
Registered: Jun 2008
Location: Västerås, Sweden
Distribution: Arch Linux, Debian, Ubuntu
Posts: 21

Original Poster
Rep: Reputation: 16
Got another problem with this, more related to crontab though.

If I add this to crontab like this :
*/30 * * * * /home/user/bash/wget.sh

Then it starts the bash script every 30 minutes but the conflict comes if the first download isn't done by the second start.

I recently had three /home/user/bash/wget.sh and three wget of the same file running at the same time.
I guess this could create a few errors in the file or just a bunch of wgets running on the same file all the time.

Anyone have any idea on how to solve this?
 
Old 06-02-2008, 07:22 AM   #7
Alien Bob
Slackware Contributor
 
Registered: Sep 2005
Location: Eindhoven, The Netherlands
Distribution: Slackware
Posts: 5,393

Rep: Reputation: Disabled
Quote:
Originally Posted by XeroXer View Post
Got another problem with this, more related to crontab though.

If I add this to crontab like this :
*/30 * * * * /home/user/bash/wget.sh

Then it starts the bash script every 30 minutes but the conflict comes if the first download isn't done by the second start.

I recently had three /home/user/bash/wget.sh and three wget of the same file running at the same time.
I guess this could create a few errors in the file or just a bunch of wgets running on the same file all the time.

Anyone have any idea on how to solve this?
The solution would be to use a lock file. The script creates lockfile when it starts, and deletes it at the end of the script. But if it finds an existing lockfile, it knows that another version of the script is still running and then it aborts.

Code:
#!/bin/sh

# Files we use:
INFILE=/home/user/urls
PIDFILE=/var/tmp/$(basename $0).pid

# Check for existing lockfile:
if [ -e $PIDFILE ]; then
  echo "Another instance (`cat $PIDFILE`) still running?"
  echo "If you are sure that no other instance is running, delete the lockfile"
  echo "'${PIDFILE}' and re-start this script."
  echo "Aborting now..."
  exit 1
else
  # Create our new lockfile:
  echo $$ > $PIDFILE
fi

for line in $(cat $INFILE | grep -v '^#'); do
  # download the file:
  wget -nv $line
  # mark the line as done by putting '#' in front:
  sed -i -e "s?^$line?#$line?" $INFILE
done

# Remove our lockfile:
rm -f $PIDFILE
 
Old 06-02-2008, 09:08 AM   #8
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,765
Blog Entries: 54

Rep: Reputation: 2976Reputation: 2976Reputation: 2976Reputation: 2976Reputation: 2976Reputation: 2976Reputation: 2976Reputation: 2976Reputation: 2976Reputation: 2976Reputation: 2976
AFAIK you don't even need a lock, there's enough envvars to go on I think: cronjobs run at a certain $SHLVL, share the $PPID ($PID of cron daemon) and if you 'pgrep -f 'cron.*/path/tojobname'' the amount of $PIDs should be exactly one plus the $PID should be equal to $$, plus fuser should show only cronjob wget process $PID accessing the wgetfile.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
bash: create a descriptor reading/writing from/to different files? Thinking Programming 0 04-19-2006 04:28 PM
writing and reading from a file! sahel Programming 1 12-27-2005 02:33 PM
BASH script – reading and writing variables to a separate file morrolan Programming 10 09-20-2005 08:45 AM
C File reading and writing AbhishekSamuel Programming 3 05-03-2005 04:59 PM
Help reading and writing to file xiste Programming 1 04-15-2005 01:43 AM


All times are GMT -5. The time now is 03:06 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration