an other BASH question. trying to put a '- ' at the front of all lines in a txt file

lleb · 07-16-2012, 09:03 AM

i am trying to take advantage of the excludes.conf file that resides in a directory we backup nightly/hourly (for rsync) to get the --exclude-from=excludes.txt. i am trying to create the excludes.txt in my rsync script.

I found a bit of code, but I was unable to make it work. mainly due to the fact I dont know enough about coding.

Code:

for excludes.txt in /usr/rx30
do
  while read -r line
  do
      echo "- ${line}"
  done < ${excludes.txt} >temp
  mv temp ${excludes.txt}
done

This provided the error on line 9, the last line of the code, excludes.txt is invalid. sorry i lost the error.

Code:

[rx30@rx30 ~]$ cat excludes.txt
*.viminfo*
*exclude*
*lifeline*.que
*lifeline*.lo*
*lifeline.sta*
*.bash_history*
*.tar*
*.bz2*
*.zip*
*.gz*
*core*
*.kde*
*.ICEauthority*
*rr_moved*
*.gnome*
*.mcop*
*.sawfish*
*.metacity*
*.nautilus*
*backup.log*
database-20120502.tar
datidx-20120521.tgz
*.bz
*.google*
*.mozilla*
*.openoffice*
*.DCOP*
*.adobe*
*.gconf*
*.thumbnail

in order to use this for rsync i would have to have a line that looks like:

Code:

 - *.bz

from what i have read.

Thanks for the help.

grail · 07-16-2012, 10:25 AM

Just to confirm, the file you have shown is the input that you wish to change to now have dashes at the front of each line?

lleb · 07-16-2012, 10:40 AM

yes that is 100% correct. in the --exclude-from=foo.txt the file format needs to be a '-' for ignore and '+' to include in the rsync.

that is if i am reading that portion of the rsync correctly.

TobiSGD · 07-16-2012, 10:45 AM

Then you did get it wrong. The exclude file has to contain the bare file/folder names, one per line, without any additional prefixes or suffixes.

lleb · 07-16-2012, 11:03 AM

correct, I understand I did it wrong with my above bit of code. Again I am trying to take the existing excludes.conf and create excludes.txt to be used with the rsync command.

The excludes.conf contains the proper list of files/types/directories I would like to 'ignore' in the rsync. The format is slightly different between the two files as well. the .conf does not require the '_' or '+' while the .txt will.

I am looking for a bit of code to take the excludes.conf mv excludes.txt and at the same time add the '- ' to the beginning of each line without the ''.

lleb · 07-16-2012, 11:05 AM

this is some of what I have read on the --exclude-from=foo

Code:

The"exclude-from" file

The exclude file is a list of directory and file names to be excluded from the rsync destination e.g.

+ /source1/.fileA
- /source1/.*
- /source1/junk/

The format of the exclude file is one pattern per line.  The pattern can be a literal string, wildcard, or character range.

In UNIX, hidden files start with a dot (.fileA is a hidden file), and * is a wildcard (.* means all hidden files).  

A leading "+" means include the pattern.  A leading "-" means exclude the pattern.

A path with a leading-slash must start with the source directory name (not the entire path).

Trailing slash is a directory (not a file).  No trailing slash can be a directory or a file.

Lines in an exclude file are read verbatim.  One frequent error is leaving some extra whitespace after a file name.

thus allowing for either a path and or a wildcard as well as hidden files.

lleb · 07-16-2012, 11:48 AM

ahh figured it out:

Code:

sed 's/^/- /' excludes.conf > excludes.txt

This did the trick:

this is the testing that i did, now i can add the little sed command to my backup scripts:

Code:

[rx30@rx30 ~]$ ls -l excl*
-rwxrwxr-x 1 rx30 group 208 Oct 21  2004 exclude.conf
-rwxrwxr-x 1 rx30 group 335 Jul  3 16:59 excludes.conf
[rx30@rx30 ~]$ chmod +x EXCLUDE
[rx30@rx30 ~]$ EXCLUDE
[rx30@rx30 ~]$ ls -l excl*
-rwxrwxr-x 1 rx30 group 208 Oct 21  2004 exclude.conf
-rwxrwxr-x 1 rx30 group 335 Jul  3 16:59 excludes.conf
-rw-rw-rw- 1 rx30 group 395 Jul 16 12:47 excludes.txt
[rx30@rx30 ~]$ cat excludes.txt
- *.viminfo*
- *exclude*
- *lifeline*.que
- *lifeline*.lo*
- *lifeline.sta*
- *.bash_history*
- *.tar*
- *.bz2*
- *.zip*
- *.gz*
- *core*
- *.kde*
- *.ICEauthority*
- *rr_moved*
- *.gnome*
- *.mcop*
- *.sawfish*
- *.metacity*
- *.nautilus*
- *backup.log*
- database-20120502.tar
- datidx-20120521.tgz
- *.bz
- *.google*
- *.mozilla*
- *.openoffice*
- *.DCOP*
- *.adobe*
- *.gconf*
- *.thumbnail
[rx30@rx30 ~]$ cat EXCLUDE
#!/bin/bash

sed 's/^/- /' excludes.conf > excludes.txt

exit;

Thanks for the help.

psyhe · 07-16-2012, 11:55 AM

Quote:

Originally Posted by lleb

ahh figured it out:

Code:

sed 's/^/- /' excludes.conf > excludes.txt

This did the trick:

You can also use simply:

Code:

sed -i 's/^/- /gm' excludes.conf

Sed will process the file inplace (-i), thus avoiding the need to move the temporary .txt file afterwards.

lleb · 07-16-2012, 01:02 PM

psyhe, are you saying it will modify the excludes.conf? i dont want that, i still need to have the output in excludes.txt as each of the two files is used for different things. the excludes.conf is for tar and the .txt is for rsync.

padeen · 07-16-2012, 07:36 PM

By the way, the original error is likely from these lines:

Code:

done < ${excludes.txt} >temp
  mv temp ${excludes.txt}

By using ${...}, you specifying a variable named excludes.txt, when you really mean just the file itself which would just be plain excludes.txt.

The error comes because the shell can't find a variable called excludes.txt.

lleb · 07-16-2012, 08:02 PM

ahh thank you padeen, that i understand. just trying to get in good coding habit and when i see $string, i automatically put {} around it.

helps to understand the difference in this case. thank you.

TobiSGD · 07-16-2012, 08:08 PM

I can only state again that this whole script is not necessary. This is my exclude-file, which I use with rsync to backup my Slackware system on my main computer:

Code:

/data
/server
/mnt
/tmp
/media
/proc
/dev
/sys
/lost+found
/win_d

You can use the notation with -/+ prefix, if you want to, but it is not necessary. You can just use plain text, respective your exclude.conf file.