LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 06-13-2014, 09:29 AM   #1
jonnybinthemix
Member
 
Registered: May 2014
Location: Bristol, United Kingdom
Distribution: RHEL 5 & 6
Posts: 169

Rep: Reputation: Disabled
Another quick bash question :)


Hey Guys

Again, another quick bash related question..

I'm using a Variable, which is associated to a file.

Code:
FILES=/tmp/files
Into which a list of filenames are written... every time the script is run it is appending the new filenames to the list within that file.

I don't want it to do that, I'd like that file only to be used when the script is running..

So, the script runs, puts the filenames into the file (variable to be called later in the script) and then the file to be wiped (but remain there) for use next time.

Any ideas?

Thanks
Jon
 
Old 06-13-2014, 09:35 AM   #2
jonnybinthemix
Member
 
Registered: May 2014
Location: Bristol, United Kingdom
Distribution: RHEL 5 & 6
Posts: 169

Original Poster
Rep: Reputation: Disabled
Argh..

I'm always conscious to make sure I've searched and tried various things before I post a question... which on this occasion I did... however I think I may have found the answer just after posting...

Code:
> $FILES
Seems to have done it..
 
Old 06-13-2014, 09:42 AM   #3
Habitual
LQ 5k Club
 
Registered: Jan 2011
Location: Nowhere near you, thank God.
Distribution: OSX Sierra
Posts: 8,573
Blog Entries: 15

Rep: Reputation: Disabled
Quote:
Originally Posted by jonnybinthemix View Post
I'm using a Variable, which is associated to a file
Jon: That's called an assignment.

Have a peek at the bash scripting references at https://www.linuxquestions.org/quest...llected-35954/
 
Old 06-13-2014, 09:44 AM   #4
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,711

Rep: Reputation: 1279Reputation: 1279Reputation: 1279Reputation: 1279Reputation: 1279Reputation: 1279Reputation: 1279Reputation: 1279Reputation: 1279
Well... actually you have to be careful.

The initial problem is that if you append the new file to the end before you finish processing everything, then you have an infinite loop (well, until you run out of disk space).

If you have a limited number of files (limited being under around 10,000), you can first load the file into an array (closing the file after loading). Then you can process the file from the array and append whatever you want to the file.

OR you can create a new empty file, and copy whichever names you need from the first file, and add new names. When finished, just "mv" the new file to the old file name.
 
Old 06-13-2014, 10:09 AM   #5
jonnybinthemix
Member
 
Registered: May 2014
Location: Bristol, United Kingdom
Distribution: RHEL 5 & 6
Posts: 169

Original Poster
Rep: Reputation: Disabled
Ah I see,

I'm now having the problem I believe..

If it helps here is the section of script that's using it:

Code:
FILES=/tmp/upload
FTP=`awk '{print "put "$0;}' $FILES`

find /sitsimp -type f -name "*$(date +%d%m%y)*.csv.gpg" | awk -F/ '{print $NF}' >> $FILES

cd /sitsimp

ftp -inv server <<!
user user pass
binary
$FTP
bye
!

> $FILES
So the variable is used to store the results from the find command, but this information is only valid until it's been uploaded by FTP. At which point the file needs to be emptied.

What would be the best way of achieving this? At the moment, with >$ FILES is does not work at all
 
Old 06-13-2014, 10:18 AM   #6
jonnybinthemix
Member
 
Registered: May 2014
Location: Bristol, United Kingdom
Distribution: RHEL 5 & 6
Posts: 169

Original Poster
Rep: Reputation: Disabled
I'm also noticing that if I empty the /tmp/upload file.. the script needs to be run twice to actually carry out the upload.

Once to write the filenames /tmp/upload and another time to upload them to the FTP Server.

Something tells me I'm probably not going about this the correct way..
 
Old 06-13-2014, 10:32 AM   #7
szboardstretcher
Senior Member
 
Registered: Aug 2006
Location: Detroit, MI
Distribution: GNU/Linux systemd
Posts: 4,211

Rep: Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612
Use the mktemp utility to create tmp files.

Code:
TMPFILE=$(mktemp /tmp/myfile.XXXXX)	
rm -f $TMPFILE
I would do something like this though... UNTESTED, but should be close.

Code:
#create the tmp file
TMPFILE=$(mktemp /tmp/myfile.XXXXX)	

#populate tmpfile
find /sitsimp -type f -name "*$(date +%d%m%y)*.csv.gpg" > $TMPFILE

cd /sitsimp

#read each line of tmp file, upload that file, move to the next
while read LINE 
do
ftp -inv server << EOF
user user pass
binary
put $LINE
bye
EOF
done < $TMPFILE

#remove tmp file
rm -f $TMPFILE
 
1 members found this post helpful.
Old 06-13-2014, 10:33 AM   #8
jonnybinthemix
Member
 
Registered: May 2014
Location: Bristol, United Kingdom
Distribution: RHEL 5 & 6
Posts: 169

Original Poster
Rep: Reputation: Disabled
Hmm.. it seems if I remove a > from the find command and insert the data into the file with a single > it over writes what's in there and does no append.

This may be okay... but still the script has to be run twice.. once to update the file and then again to upload the data.
 
Old 06-13-2014, 10:35 AM   #9
jonnybinthemix
Member
 
Registered: May 2014
Location: Bristol, United Kingdom
Distribution: RHEL 5 & 6
Posts: 169

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by szboardstretcher View Post
Use the mktemp utility to create tmp files.

Code:
TMPFILE=$(mktemp /tmp/myfile.XXXXX)	
rm -f $TMPFILE
I would do something like this though... UNTESTED, but should be close.

Code:
#create the tmp file
TMPFILE=$(mktemp /tmp/myfile.XXXXX)	

#populate tmpfile
find /sitsimp -type f -name "*$(date +%d%m%y)*.csv.gpg" > $TMPFILE

cd /sitsimp

#read each line of tmp file, upload that file, move to the next
while read LINE 
do
ftp -inv server << EOF
user user pass
binary
put $LINE
bye
EOF
done < $TMPFILE

#remove tmp file
rm -f $TMPFILE
Hi,

Thanks for your message.

I've been playing around with the while loop all day and I've been pulling my hair out.. couldn't get it to work.

I'll have a play around with what you've suggested and see if I have more luck

Will update..

Thanks
 
Old 06-16-2014, 10:23 AM   #10
jonnybinthemix
Member
 
Registered: May 2014
Location: Bristol, United Kingdom
Distribution: RHEL 5 & 6
Posts: 169

Original Poster
Rep: Reputation: Disabled
Hey Pal,

Thanks for your help with the FTP Loop thing, that works nice It seems when I was trying to get it working I didn't put the;

Code:
done < $TMPFILE
Does this tell it that it's only done when it's read through all lines within that file?

The following is my now (nearly) completed script... Is it possible for me to consolodate some of the loops? I've tried renaming the file extension in the same loop as encrypting the, but it doesn't work and only renames the unencrypted file.

Code:
#!/bin/bash

TMPFILE=$(mktemp /tmp/tempfile1.XXXXX)
HOST=X.X.X.X
USER=ftpxfer
PASS=xxxx
FTPLOG=/tmp/ftplogfile

cd /sitsimp

find . -type f -name "*$(date +%d%m%y)*.csv" | 
	while read i; do
			gpg -r server@bathspa.ac.uk -e "$i";
	done

find . -type f -name "*$(date +%d%m%y)*.csv.gpg" |
	while read i; do
		mv $i `basename $i .csv.gpg`.csv.pgp;
	done	

find . -type f -name "*$(date +%d%m%y)*.csv.pgp" | 

awk -F/ '{print $NF}' > $TMPFILE

read LINE
	do
		ftp -inv $HOST << EOF > $FTPLOG
		user $USER xxxx 
		binary
		put $LINE
		bye
EOF

fgrep "226 Transfer complete" $FTPLOG ; then
        		echo -e "FTP TRANSFER SUCCESS.\n\n $LINE has uploaded to Server\n"
		else	
       			 echo -e "FTP TRANSFER ERROR.\n\n $LINE failed to upload\n"
fi
done < $TMPFILE

rm -f $TMPFILE
 
Old 06-16-2014, 01:03 PM   #11
szboardstretcher
Senior Member
 
Registered: Aug 2006
Location: Detroit, MI
Distribution: GNU/Linux systemd
Posts: 4,211

Rep: Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612
No problem Chief,

Looks like you have a space in there:

Code:
mv $i `basename $i .csv.gpg`.csv.pgp;
Between the $i and .csv.gpg. If you remove that and combine the while loops you should be ok.
 
1 members found this post helpful.
Old 06-16-2014, 01:21 PM   #12
jonnybinthemix
Member
 
Registered: May 2014
Location: Bristol, United Kingdom
Distribution: RHEL 5 & 6
Posts: 169

Original Poster
Rep: Reputation: Disabled
Ah yes, I see that... although it's strange because the script is working well lol.

Does it only make a difference in combining the loops?
 
Old 06-16-2014, 01:42 PM   #13
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Debian, Arch
Posts: 3,239

Rep: Reputation: 1405Reputation: 1405Reputation: 1405Reputation: 1405Reputation: 1405Reputation: 1405Reputation: 1405Reputation: 1405Reputation: 1405Reputation: 1405
Space is correct:
Code:
$ i=foo.csv.gpg
$ basename $i.csv.gpg
foo.csv.gpg.csv.gpg
$ basename $i .csv.gpg
foo
I recommend using -o option to gpg, to get the name right the first time:
Code:
find . -type f -name "*$(date +%d%m%y)*.csv" | 
	while read i; do
			gpg -r server@bathspa.ac.uk -e "$i" -o "$i.pgp";
	done
 
2 members found this post helpful.
Old 06-16-2014, 01:57 PM   #14
jonnybinthemix
Member
 
Registered: May 2014
Location: Bristol, United Kingdom
Distribution: RHEL 5 & 6
Posts: 169

Original Poster
Rep: Reputation: Disabled
Ahh.. now that's a better plan.

I had thought about a switch to do it inline, but didn't think it would be as simple as $i.pgp!!

I like it

Do you think the rest of the script is okay? I am still learning BASH and broke the script up into individual jobs, worked out how to do that particular bit, and then added it all together I'm pleased with what I've learned but always want to learn more

Ooh, and, was I correct about the:

done < $TMPFILE

Does that mean that, the loop is only done when the end of $TMPFILE has been reached?
 
Old 06-16-2014, 02:58 PM   #15
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Debian, Arch
Posts: 3,239

Rep: Reputation: 1405Reputation: 1405Reputation: 1405Reputation: 1405Reputation: 1405Reputation: 1405Reputation: 1405Reputation: 1405Reputation: 1405Reputation: 1405
Quote:
Originally Posted by jonnybinthemix View Post
done < $TMPFILE

Does that mean that, the loop is only done when the end of $TMPFILE has been reached?
Sort of. It means all the commands inside the while loop have their standard input redirected from $TMPFILE. The while read LINE part means to read from input ($TMPFILE because of the redirection) and stop when end of file is reached.


All the input files are expected to be in the same directory, right? That is, you don't expect to find files in subdirectories (I think the basename thing wouldn't work otherwise)? If that's so you can replace the find | while with globbing in a for loop:

Code:
for i in *"$(date +%d%m%y)"*.csv; do
    gpg -r server@bathspa.ac.uk -e "$i" -o "$i.pgp";
done
And you could combine the ftp loop contents into this one and avoid $TEMPFILE altogether:


Code:
for i in *"$(date +%d%m%y)"*.csv; do
    gpg -r server@bathspa.ac.uk -e "$i" -o "$i.pgp";

    ftp -inv $HOST << EOF > $FTPLOG
user $USER xxxx 
binary
put $i.pgp
bye
EOF

    if fgrep "226 Transfer complete" $FTPLOG ; then
       echo -e "FTP TRANSFER SUCCESS.\n\n $i.pgp has uploaded to Server\n"
    else	
       echo -e "FTP TRANSFER ERROR.\n\n $i.pgp failed to upload\n"
    fi
done

Last edited by ntubski; 06-16-2014 at 04:03 PM. Reason: forgot to replace $LINE with $i.pgp
 
1 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Quick BASH Question jonnybinthemix Linux - Newbie 2 06-13-2014 08:55 AM
Quick Bash question for a hosting company webjive Linux - Server 13 12-01-2011 07:47 PM
A quick bash question Ander Linux - Newbie 6 05-05-2006 03:13 AM
Quick bash question neocytrix Programming 8 07-30-2004 11:23 PM
really quick bash question fibbi Linux - Software 3 06-15-2004 11:14 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 10:38 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration