LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices



Reply
 
Search this Thread
Old 07-16-2012, 10:03 AM   #1
lleb
Senior Member
 
Registered: Dec 2005
Location: Florida
Distribution: CentOS/Fedora
Posts: 2,582

Rep: Reputation: 480Reputation: 480Reputation: 480Reputation: 480Reputation: 480
an other BASH question. trying to put a '- ' at the front of all lines in a txt file


i am trying to take advantage of the excludes.conf file that resides in a directory we backup nightly/hourly (for rsync) to get the --exclude-from=excludes.txt. i am trying to create the excludes.txt in my rsync script.

I found a bit of code, but I was unable to make it work. mainly due to the fact I dont know enough about coding.

Code:
for excludes.txt in /usr/rx30
do
  while read -r line
  do
      echo "- ${line}"
  done < ${excludes.txt} >temp
  mv temp ${excludes.txt}
done

This provided the error on line 9, the last line of the code, excludes.txt is invalid. sorry i lost the error.

Code:
[rx30@rx30 ~]$ cat excludes.txt
*.viminfo*
*exclude*
*lifeline*.que
*lifeline*.lo*
*lifeline.sta*
*.bash_history*
*.tar*
*.bz2*
*.zip*
*.gz*
*core*
*.kde*
*.ICEauthority*
*rr_moved*
*.gnome*
*.mcop*
*.sawfish*
*.metacity*
*.nautilus*
*backup.log*
database-20120502.tar
datidx-20120521.tgz
*.bz
*.google*
*.mozilla*
*.openoffice*
*.DCOP*
*.adobe*
*.gconf*
*.thumbnail
in order to use this for rsync i would have to have a line that looks like:
Code:
 - *.bz
from what i have read.

Thanks for the help.
 
Old 07-16-2012, 11:25 AM   #2
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,692

Rep: Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987Reputation: 1987
Just to confirm, the file you have shown is the input that you wish to change to now have dashes at the front of each line?
 
Old 07-16-2012, 11:40 AM   #3
lleb
Senior Member
 
Registered: Dec 2005
Location: Florida
Distribution: CentOS/Fedora
Posts: 2,582

Original Poster
Rep: Reputation: 480Reputation: 480Reputation: 480Reputation: 480Reputation: 480
yes that is 100% correct. in the --exclude-from=foo.txt the file format needs to be a '-' for ignore and '+' to include in the rsync.

that is if i am reading that portion of the rsync correctly.
 
Old 07-16-2012, 11:45 AM   #4
TobiSGD
Moderator
 
Registered: Dec 2009
Location: Hanover, Germany
Distribution: Main: Gentoo Others: What fits the task
Posts: 15,651
Blog Entries: 2

Rep: Reputation: 4095Reputation: 4095Reputation: 4095Reputation: 4095Reputation: 4095Reputation: 4095Reputation: 4095Reputation: 4095Reputation: 4095Reputation: 4095Reputation: 4095
Then you did get it wrong. The exclude file has to contain the bare file/folder names, one per line, without any additional prefixes or suffixes.
 
Old 07-16-2012, 12:03 PM   #5
lleb
Senior Member
 
Registered: Dec 2005
Location: Florida
Distribution: CentOS/Fedora
Posts: 2,582

Original Poster
Rep: Reputation: 480Reputation: 480Reputation: 480Reputation: 480Reputation: 480
correct, I understand I did it wrong with my above bit of code. Again I am trying to take the existing excludes.conf and create excludes.txt to be used with the rsync command.

The excludes.conf contains the proper list of files/types/directories I would like to 'ignore' in the rsync. The format is slightly different between the two files as well. the .conf does not require the '_' or '+' while the .txt will.

I am looking for a bit of code to take the excludes.conf mv excludes.txt and at the same time add the '- ' to the beginning of each line without the ''.
 
Old 07-16-2012, 12:05 PM   #6
lleb
Senior Member
 
Registered: Dec 2005
Location: Florida
Distribution: CentOS/Fedora
Posts: 2,582

Original Poster
Rep: Reputation: 480Reputation: 480Reputation: 480Reputation: 480Reputation: 480
this is some of what I have read on the --exclude-from=foo

Code:
The"exclude-from" file

The exclude file is a list of directory and file names to be excluded from the rsync destination e.g.

+ /source1/.fileA
- /source1/.*
- /source1/junk/

The format of the exclude file is one pattern per line.  The pattern can be a literal string, wildcard, or character range.

In UNIX, hidden files start with a dot (.fileA is a hidden file), and * is a wildcard (.* means all hidden files).  

A leading "+" means include the pattern.  A leading "-" means exclude the pattern.

A path with a leading-slash must start with the source directory name (not the entire path).

Trailing slash is a directory (not a file).  No trailing slash can be a directory or a file.

Lines in an exclude file are read verbatim.  One frequent error is leaving some extra whitespace after a file name.
thus allowing for either a path and or a wildcard as well as hidden files.
 
Old 07-16-2012, 12:48 PM   #7
lleb
Senior Member
 
Registered: Dec 2005
Location: Florida
Distribution: CentOS/Fedora
Posts: 2,582

Original Poster
Rep: Reputation: 480Reputation: 480Reputation: 480Reputation: 480Reputation: 480
ahh figured it out:

Code:
sed 's/^/- /' excludes.conf > excludes.txt
This did the trick:

this is the testing that i did, now i can add the little sed command to my backup scripts:

Code:
[rx30@rx30 ~]$ ls -l excl*
-rwxrwxr-x 1 rx30 group 208 Oct 21  2004 exclude.conf
-rwxrwxr-x 1 rx30 group 335 Jul  3 16:59 excludes.conf
[rx30@rx30 ~]$ chmod +x EXCLUDE
[rx30@rx30 ~]$ EXCLUDE
[rx30@rx30 ~]$ ls -l excl*
-rwxrwxr-x 1 rx30 group 208 Oct 21  2004 exclude.conf
-rwxrwxr-x 1 rx30 group 335 Jul  3 16:59 excludes.conf
-rw-rw-rw- 1 rx30 group 395 Jul 16 12:47 excludes.txt
[rx30@rx30 ~]$ cat excludes.txt
- *.viminfo*
- *exclude*
- *lifeline*.que
- *lifeline*.lo*
- *lifeline.sta*
- *.bash_history*
- *.tar*
- *.bz2*
- *.zip*
- *.gz*
- *core*
- *.kde*
- *.ICEauthority*
- *rr_moved*
- *.gnome*
- *.mcop*
- *.sawfish*
- *.metacity*
- *.nautilus*
- *backup.log*
- database-20120502.tar
- datidx-20120521.tgz
- *.bz
- *.google*
- *.mozilla*
- *.openoffice*
- *.DCOP*
- *.adobe*
- *.gconf*
- *.thumbnail
[rx30@rx30 ~]$ cat EXCLUDE
#!/bin/bash

sed 's/^/- /' excludes.conf > excludes.txt

exit;
Thanks for the help.
 
Old 07-16-2012, 12:55 PM   #8
psyhe
LQ Newbie
 
Registered: Mar 2006
Posts: 5

Rep: Reputation: 1
Quote:
Originally Posted by lleb View Post
ahh figured it out:

Code:
sed 's/^/- /' excludes.conf > excludes.txt
This did the trick:
You can also use simply:
Code:
sed -i 's/^/- /gm' excludes.conf
Sed will process the file inplace (-i), thus avoiding the need to move the temporary .txt file afterwards.
 
Old 07-16-2012, 02:02 PM   #9
lleb
Senior Member
 
Registered: Dec 2005
Location: Florida
Distribution: CentOS/Fedora
Posts: 2,582

Original Poster
Rep: Reputation: 480Reputation: 480Reputation: 480Reputation: 480Reputation: 480
psyhe, are you saying it will modify the excludes.conf? i dont want that, i still need to have the output in excludes.txt as each of the two files is used for different things. the excludes.conf is for tar and the .txt is for rsync.
 
Old 07-16-2012, 08:36 PM   #10
padeen
Member
 
Registered: Sep 2009
Location: Perth, W.A.
Distribution: Slackware 14, Debian 7, FreeBSD, OpenBSD
Posts: 182

Rep: Reputation: 35
By the way, the original error is likely from these lines:
Code:
done < ${excludes.txt} >temp
  mv temp ${excludes.txt}
By using ${...}, you specifying a variable named excludes.txt, when you really mean just the file itself which would just be plain excludes.txt.

The error comes because the shell can't find a variable called excludes.txt.
 
1 members found this post helpful.
Old 07-16-2012, 09:02 PM   #11
lleb
Senior Member
 
Registered: Dec 2005
Location: Florida
Distribution: CentOS/Fedora
Posts: 2,582

Original Poster
Rep: Reputation: 480Reputation: 480Reputation: 480Reputation: 480Reputation: 480
ahh thank you padeen, that i understand. just trying to get in good coding habit and when i see $string, i automatically put {} around it.

helps to understand the difference in this case. thank you.
 
Old 07-16-2012, 09:08 PM   #12
TobiSGD
Moderator
 
Registered: Dec 2009
Location: Hanover, Germany
Distribution: Main: Gentoo Others: What fits the task
Posts: 15,651
Blog Entries: 2

Rep: Reputation: 4095Reputation: 4095Reputation: 4095Reputation: 4095Reputation: 4095Reputation: 4095Reputation: 4095Reputation: 4095Reputation: 4095Reputation: 4095Reputation: 4095
I can only state again that this whole script is not necessary. This is my exclude-file, which I use with rsync to backup my Slackware system on my main computer:
Code:
/data
/server
/mnt
/tmp
/media
/proc
/dev
/sys
/lost+found
/win_d
You can use the notation with -/+ prefix, if you want to, but it is not necessary. You can just use plain text, respective your exclude.conf file.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
A script that removes dubblets lines into a TXT file? frenchn00b Programming 17 11-20-2012 01:04 PM
cut first 10 lines of file master.txt and paste in ab1.txt and so on yogeshkumkar Programming 4 08-31-2011 08:23 AM
[SOLVED] Using AWK to print out the first few lines of a txt file mskalak Linux - Newbie 3 07-27-2011 03:58 PM
Basic bash script question re: file size or # of lines in a file the_fornicator Programming 6 09-03-2009 10:41 AM
Finding things on lines of a txt file Jayfrin Programming 3 05-15-2008 09:13 AM


All times are GMT -5. The time now is 01:12 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration