LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 04-23-2011, 02:38 AM   #1
wonderfullyrich
LQ Newbie
 
Registered: Sep 2009
Location: Kampala
Distribution: Ubuntu
Posts: 10

Rep: Reputation: 0
In a bash script, how do I move a line to a new file.


I've got a bash script I'm using to download a text file list of links via axel. What I'd like to do is automate the movement of completed links in the for loop when axel has successfully completed the download.

This is what I've got. I can figure that I can just echo append the line to a new file, but what is the easiest way to delete the line with the link I just downloaded?


Code:
#!/bin/bash

for i in $( cat $1); do
axel --alternate --num-connections=6 $i
export RC=$?
	if [ "$RC" = "0" ]; then
	   echo 
	   echo Succeeded downloading following link $i 
	   echo $i >> downloaded-links.txt
	   echo 
		#remove $i line from $1 file...?
	else
	    echo 
	    echo Failed downloading following link $i 
	    echo $i >> failed-links.txt
	    echo 
	fi

done
 
Old 04-23-2011, 02:49 AM   #2
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371
Hi,

Assuming that all lines are unique:
Code:
#remove $i line from $1 file...?
sed -i "/$i/d" $1
Hope this helps.
 
1 members found this post helpful.
Old 04-23-2011, 03:09 AM   #3
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,431

Rep: Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878
You do also know that your export line is not required. In fact, you could just use the return value on its own:
Code:
if (( $? ))
The only difference is you need to change the logic as 0 will now be false so change then and else sections around.
 
Old 04-23-2011, 08:05 AM   #4
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Debian
Posts: 2,396

Rep: Reputation: 814Reputation: 814Reputation: 814Reputation: 814Reputation: 814Reputation: 814Reputation: 814
You don't even need to look at the return value at all, if already does that:
Code:
if axel --alternate --num-connections=6 $i ; then
   # success
else
   # failure
fi
 
Old 04-23-2011, 09:58 AM   #5
konsolebox
Senior Member
 
Registered: Oct 2005
Distribution: Gentoo, Slackware, LFS
Posts: 2,245
Blog Entries: 15

Rep: Reputation: 233Reputation: 233Reputation: 233
just an example. i would do things this way.

Code:
#!/bin/bash

remove=()
j=0

while read i; do
    (( j++ ))

    if axel --alternate --num-connections=6 "$i"; then
        echo
        echo "Succeeded downloading following link $i"
        echo "$i" >> downloaded-links.txt
        echo
        remove=("${remove[@]}" "-e" "${j}d")
    else
        echo
        echo "Failed downloading following link $i"
        echo "$i" >> failed-links.txt
        echo
    fi
done < "$1"

sed -i "${remove[@]}" "$1"
 
1 members found this post helpful.
Old 04-25-2011, 11:27 PM   #6
wonderfullyrich
LQ Newbie
 
Registered: Sep 2009
Location: Kampala
Distribution: Ubuntu
Posts: 10

Original Poster
Rep: Reputation: 0
I just had a chance to change up the script and try it out, and I now see the duplication of the return value and indeed I see the improve simplicity of konsolebox's example script. Thanks guys!

One problem that I didn't anticipate was that although the example script does work, it's doesn't initiate the sed change till after all the links are downloaded, so if I break the script during download (or during the while function) and restart it start's over at the top of the list.

This is partially due to the fact that I reorder the links due to change in priorities. Normally I just break the script and restart it to affect that change in priorities, but is there a way to create a loop that 1. when it's finished downloading a link it removes the line, 2. actively reads the file during every loop and initiates the download of the top line. 3. stops at the end of the file?

I started on this idea below, but am not well versed enough in sed, loops, and bash to make it work. Specifically I'm struggling with the sed line as the links are fully qualified urls so I can't use / for a delimiter.

Code:
#!/bin/bash

j=0

while [ -s "$1" ]; do
	if [ $j -le 0 ]; then
		(( j++ ))
		i=$(head -1 $1)

		    if axel --alternate --num-connections=6 "$i"; then
			echo
			echo "Succeeded downloading following link $i"
			echo "$i" >> downloaded-links.txt
			echo
#This isn't working... but it's an idea.
        		remove=("|$i|d")
			sed -i '$remove' $1
		    else
			echo
			echo "Failed downloading following link $i"
			echo "$i" >> failed-links.txt
			echo
		    fi
	else
		(( j=0))
	fi

done
 
Old 04-26-2011, 11:36 AM   #7
konsolebox
Senior Member
 
Registered: Oct 2005
Distribution: Gentoo, Slackware, LFS
Posts: 2,245
Blog Entries: 15

Rep: Reputation: 233Reputation: 233Reputation: 233
I can't test this but I hope it works for you.

Code:
#!/bin/bash

FILE=$1
TOTALLINES=$(exec wc -l "$FILE")
LINENUMBER=1

for (( I = 1; I <= TOTALLINES; I++ )); do
	if read < <(exec sed -n "${LINENUMBER}p" "$FILE"); then
		if [[ -n $REPLY ]]; then
			if axel --alternate --num-connections=6 "$REPLY"; then
				echo
				echo "Succeeded downloading following link $REPLY"
				echo "$REPLY" >> downloaded-links.txt
				echo
				sed -i "${LINENUMBER}d" "$FILE"
				continue
			else
				echo
				echo "Failed downloading following link $REPLY"
				echo "$REPLY" >> failed-links.txt
				echo
			fi
		fi
	fi

	(( LINENUMBER++ ))
done

Last edited by konsolebox; 04-26-2011 at 11:38 AM.
 
Old 04-26-2011, 12:07 PM   #8
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,431

Rep: Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878
I am not sure I see how the for loop construct would be better than a simple while loop reading from the file? (of course not to say it doesn't work)

@OP - I am not sure how breaking the script at the point of using axel makes a difference if after each successful download you are removing the lines
from the file? Obviously if you cancel the script at any point and a download is incomplete then the corresponding entry will not have been removed so the download
will start afresh once the script is started again.

Maybe I am missing something but is it not enough to say:
Code:
#!/bin/bash

FILE="$1"

while read -r LINE
do
    if axel --alternate --num-connections=6 "$LINE"; then
	echo
	echo "Succeeded downloading following link $LINE"
	echo "$LINE" >> downloaded-links.txt
	echo
	sed -i '1d' "$FILE"
    else
	echo
	echo "Failed downloading following link $LINE"
	echo "$LINE" >> failed-links.txt
	echo
    fi
done<"$FILE"
 
Old 04-27-2011, 12:08 AM   #9
wonderfullyrich
LQ Newbie
 
Registered: Sep 2009
Location: Kampala
Distribution: Ubuntu
Posts: 10

Original Poster
Rep: Reputation: 0
Thanks all. Tis all much appreciated.

To answer your question grail, ideally i'd just do a loop based on reading from the file, however I edit the url file on the fly in the background and so the number of lines change in the file up and down depending. If I use a loop to read the file, it's retains the order and number without integrating new lines, deleted lines, or order changes, which is why I was trying to figure out how to make the loop dependent on the end of file. This also makes any attempt at deleting the line based solely on the line number precarious, as it could have changed, so that's why I was trying to delete the actual entry rather then the line number or number of lines.

I think this answers your second question too, but I thought I'd say that yes axel and the script does just leave the incomplete download listed. The reason I move it out is because if axel is given a download and finds a dup file without state information (i.e. an .st file), it restarts the download from scratch with the .0 extension (and increments). This means if it's completed the download of a file (and there's no longer an .st file), it'll just keep downloading the first link on the list adding new extension even if the last file has been successfully completed. It's not quite as smart as wget with no clobber, but the multi-connection is more useful in this particular case.

Any ideas? Was my loop just missing the sed delimiter, or was the whole loop entirely malformed?

Many thanks guys!
Rich
 
Old 04-27-2011, 01:28 AM   #10
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,431

Rep: Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878
Code:
sed -i '$remove' $1
This does not work as the single quotes have stopped the shell from expanding the variable so you should use double quotes
to get around this.

Now that you have explained further I do understand what you intend but would caution you that it is indeed fraught with danger.
Even though you only access the file using head to get an entry the big problems will occur while you are changing the file
and head tries to grab the next line. Then at the same time if the download is quick you may also execute a sed on the file whilst
you are still editing. All of this chills me to the bone as data will actively be added and deleted at the same time. Sounds
like a recipe for disaster.

The only thing springing to mind would be to create a lock on the file so when one or the other application is making changes the other
one, you or script, will have to wait until it is freed.

I'll be interested to see how this is solved??
 
Old 04-27-2011, 05:27 AM   #11
konsolebox
Senior Member
 
Registered: Oct 2005
Distribution: Gentoo, Slackware, LFS
Posts: 2,245
Blog Entries: 15

Rep: Reputation: 233Reputation: 233Reputation: 233
you can try this one. it requires bash version 4.0 or newer.
Code:
#!/bin/bash

FILE=$1
declare -A SUCCESS=()
declare -A FAILED=()

for (( ;; )); do
	ACTIVE=false

	for (( ;; )); do
		if read; do
			if [[ -z $REPLY ]]; then
				echo >&3
				continue
			elif [[ -n ${SUCCESS[$REPLY]} ]]; then
				continue
			elif [[ -n ${FAILED[$REPLY]} ]]; then
				echo "$REPLY" >&3
				continue
			elif axel --alternate --num-connections=6 "$REPLY"; then
				echo
				echo "Succeeded downloading following link $REPLY"
				echo "$REPLY" >> downloaded-links.txt
				echo
				SUCCESS[$REPLY]=.
			else
				echo
				echo "Failed downloading following link $REPLY"
				echo "$REPLY" >> failed-links.txt
				echo "$REPLY" >&3
				echo
				FAILED[$REPLY]=.
			fi

			while read; do
				[[ -z ${SUCCESS[$REPLY]} ]] && echo "$REPLY" >&3
			done

			ACTIVE=true
		fi

		break
	done < "$FILE" 3>temp.txt

	cat tempt.txt > "$FILE"

	[[ $ACTIVE = true ]] || break
done

Last edited by konsolebox; 04-27-2011 at 05:29 AM.
 
Old 04-27-2011, 06:25 AM   #12
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,431

Rep: Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878
@konsolebox - whilst an interesting solution, how does this address the issues of the FILE being edited whilst, for example, you are catting the temp.txt back over the same FILE?

I am not saying this won't work, I am more just curious how this may / will stop this from occurring?
 
Old 04-28-2011, 02:24 AM   #13
konsolebox
Senior Member
 
Registered: Oct 2005
Distribution: Gentoo, Slackware, LFS
Posts: 2,245
Blog Entries: 15

Rep: Reputation: 233Reputation: 233Reputation: 233
sorry i had to revise it. this is the best solution i know so far since i can't consider file locking yet:

Code:
#!/bin/bash

FILE=$1
declare -A SUCCESS=()
declare -A FAILED=()

for (( ;; )); do
	# find a new link

	cat "$FILE" > temp.txt

	HASNEW=false

	while read; do
		[[ -z $REPLY || -n ${SUCCESS[$REPLY]} || -n ${FAILED[$REPLY]} ]] && continue
		HASNEW=true
		break
	done < temp.txt

	[[ $HASNEW = true ]] || break

	# download

	if axel --alternate --num-connections=6 "$REPLY"; then
		echo
		echo "Succeeded downloading following link $REPLY"
		echo "$REPLY" >> downloaded-links.txt
		echo
		SUCCESS[$REPLY]=.
	else
		echo
		echo "Failed downloading following link $REPLY"
		echo "$REPLY" >> failed-links.txt
		echo
		FAILED[$REPLY]=.
	fi

	# refresh file

	cat "$FILE" > temp.txt

	while read; do
		[[ -z ${SUCCESS[REPLY]} ]] && echo "$REPLY"
	done < temp.txt > "$FILE"
done
@grail: i think of the solution as something that could work with a file monitor like a looping "cat" or an editor that automatically detects file changes

Last edited by konsolebox; 04-28-2011 at 03:45 AM.
 
Old 04-28-2011, 03:48 AM   #14
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,431

Rep: Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878Reputation: 1878
hmmm ... so you have effectively replaced sed work with a while loop ... my thought here is that this would probably take longer to process than sed and lead more to
the issue of not locking the file (maybe)

Ignoring this constant issue, I am curious about this portion of code:
Code:
while read; do
    [[ -z $REPLY || -n ${SUCCESS[$REPLY]} || -n ${FAILED[$REPLY]} ]] && continue
    HASNEW=true
    break
done < temp.txt
I am not sure of the relevance of the null tests?

By virtue of the last while loop in the code, all not null SUCCESSes will be removed, yes?
And the FAILED items, do we not wish to retry these?

Maybe I am following it wrong ... sorry
 
Old 04-28-2011, 04:09 AM   #15
konsolebox
Senior Member
 
Registered: Oct 2005
Distribution: Gentoo, Slackware, LFS
Posts: 2,245
Blog Entries: 15

Rep: Reputation: 233Reputation: 233Reputation: 233
Quote:
Originally Posted by grail View Post
hmmm ... so you have effectively replaced sed work with a while loop ... my thought here is that this would probably take longer to process than sed and lead more to
the issue of not locking the file (maybe)
I actually agree a bit about the longer process.. it's actually the obvious disadvantage.. not unless we use a faster scripting language that can manipulate external programs too.

Quote:
Originally Posted by grail View Post
I am not sure of the relevance of the null tests?

By virtue of the last while loop in the code, all not null SUCCESSes will be removed, yes?
And the FAILED items, do we not wish to retry these?

Maybe I am following it wrong ... sorry
I just want to make the one editing the file not lose positions in lines as much as possible when editing. With the last loop, only the ones that were successfully downloaded are the ones that are *excluded. They are checked again and again since we'll never know if the one editing the file will be able to override the new version of the file with every save in the editor.

Btw, it's ok

Last edited by konsolebox; 04-28-2011 at 04:24 AM.
 
  


Reply

Tags
bash scripting


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
bash shell script read file line by line. Darren[UoW] Programming 51 07-04-2013 08:16 PM
BASH script to move files/folders to either the .Trash file or to another direcotry bettsy583 Linux - Newbie 13 12-03-2010 03:41 AM
Bash script only appears to process one line of file Nylex Programming 5 08-19-2007 10:35 PM
bash script for displaying a file one line at a time. panchosansa Programming 6 10-12-2006 10:54 AM


All times are GMT -5. The time now is 06:29 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration