[SOLVED] Trying to tar from files listed in .txt file

Batistuta_g_2000 · 03-09-2013, 09:55 AM

Hi,

I am trying to tar files from a list of files
using -T option - but is saying cannot find - is ~/ a problem? some exist and others do not - buit saying none do??

ubuntu@ubuntu:~$ tar -czvf ~/assignment1/backups/test.tar.gz -T "$list"
tar: # IMPORTANT FILES DIRECTORIES OR FILES : Cannot stat: No such file or directory
tar: ~/assignment1/important_files: Cannot stat: No such file or directory
tar: ~/assignment/important_files/file1: Cannot stat: No such file or directory
tar: ~/assignment/important_files/file2: Cannot stat: No such file or directory
tar: ~/assignment/important_files/file3: Cannot stat: No such file or directory
tar: ~/assignment/important_files/file4: Cannot stat: No such file or directory
tar: ~/assignment/important_files/file5: Cannot stat: No such file or directory
tar: ~/assginment1/important_files/file9: Cannot stat: No such file or directory
tar: ~/assignment1/important_files/file1: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors

ubuntu@ubuntu:~$ cat $list
# IMPORTANT FILES DIRECTORIES OR FILES
~/assignment1/important_files
~/assignment/important_files/file1
~/assignment/important_files/file2
~/assignment/important_files/file3
~/assignment/important_files/file4
~/assignment/important_files/file5
~/assginment1/important_files/file9
~/assignment1/important_files/file1

shivaa · 03-09-2013, 10:12 AM

First thing is, $list should contain only filenames. So remove this part:

Code:

# IMPORTANT FILES DIRECTORIES OR FILES

Also why are you making tar of files inside ~/assignment1/important_files directory when you're archiving parant dir. itself?

Code:

~$ tar -czvf ~/assignment1/backups/test.tar.gz ~/assignment1/important_files

So remove it's entry from the list as well or remove all it's child file entries from $list. Also did you check those file really exist or not? If exist, then replace ~ with /home/username. New list should look like:

Code:

buntu@ubuntu:~$ cat list
/home/username/assignment1/important_files
/home/username/assignment/important_files/file1
/home/username/assignment/important_files/file2
/home/username/assignment/important_files/file3
/home/username/assignment/important_files/file4
/home/username/assignment/important_files/file5
/home/username/assginment1/important_files/file9
/home/username/assignment1/important_files/file1

One more thing, is all this part of a script? If not, then use simple filename i.e. list, not $list.

Code:

ubuntu@ubuntu:~$ tar -czvf ~/assignment1/backups/test.tar.gz -T list

Batistuta_g_2000 · 03-09-2013, 11:04 AM

Quote:

Originally Posted by shivaa

First thing is, $list should contain only filenames. So remove this part:

Code:

# IMPORTANT FILES DIRECTORIES OR FILES

Also why are you making tar of files inside ~/assignment1/important_files directory when you're archiving parant dir. itself?

Code:

~$ tar -czvf ~/assignment1/backups/test.tar.gz ~/assignment1/important_files

So remove it's entry from the list as well or remove all it's child file entries from $list. Also did you check those file really exist or not? If exist, then replace ~ with /home/username. New list should look like:

Code:

buntu@ubuntu:~$ cat list
/home/username/assignment1/important_files
/home/username/assignment/important_files/file1
/home/username/assignment/important_files/file2
/home/username/assignment/important_files/file3
/home/username/assignment/important_files/file4
/home/username/assignment/important_files/file5
/home/username/assginment1/important_files/file9
/home/username/assignment1/important_files/file1

One more thing, is all this part of a script? If not, then use simple filename i.e. list, not $list.

Code:

ubuntu@ubuntu:~$ tar -czvf ~/assignment1/backups/test.tar.gz -T list

Thanks very much was lots wrong there, It is part of a script, I fixed them and it works but is backing up to backups and when I extract it has a number of sub-directories:

home/ubuntu/assignment1/backups/home/ubuntu/assignment1/important_files$

What I really wanted is just some of the important files from important_files (listed in $list) to be put in test.tar.gz, when unzipped (xvzf) I just want see is these files extracted in:

home/ubuntu/assignment1/backups/important_files/file1....file2....etc

shivaa · 03-09-2013, 11:45 AM

Then your script should contain following snippet:

Code:

#!/bin/bash
........
........
list=/path/to/list.txt
tar -czvf /home/ubuntu/assignment1/backups/test.tar.gz -T $list
.......
.......

Or better split it in two parts, as:

Code:

#!/bin/bash
........
........
list=/path/to/list.txt
tar -cvf /home/ubuntu/assignment1/backups/test.tar -T $list
gzip /home/ubuntu/assignment1/backups/test.tar
.......
.......

And prepare your list file careful with absolute path of files only, like:

Code:

ubuntu@ubuntu:~$ cat list
/home/username/assignment1/important_files
/home/username/assignment/important_files/file1
/home/username/assignment/important_files/file2
/home/username/assignment/important_files/file3
/home/username/assignment/important_files/file4
/home/username/assignment/important_files/file5
/home/username/assginment1/important_files/file9
/home/username/assignment1/important_files/file1

Batistuta_g_2000 · 03-10-2013, 11:47 AM

Quote:

Originally Posted by shivaa

Then your script should contain following snippet:

Code:

#!/bin/bash
........
........
list=/path/to/list.txt
tar -czvf /home/ubuntu/assignment1/backups/test.tar.gz -T $list
.......
.......

Or better split it in two parts, as:

Code:

#!/bin/bash
........
........
list=/path/to/list.txt
tar -cvf /home/ubuntu/assignment1/backups/test.tar -T $list
gzip /home/ubuntu/assignment1/backups/test.tar
.......
.......

And prepare your list file careful with absolute path of files only, like:

Code:

ubuntu@ubuntu:~$ cat list
/home/username/assignment1/important_files
/home/username/assignment/important_files/file1
/home/username/assignment/important_files/file2
/home/username/assignment/important_files/file3
/home/username/assignment/important_files/file4
/home/username/assignment/important_files/file5
/home/username/assginment1/important_files/file9
/home/username/assignment1/important_files/file1

Going blind at this stage, have my function to backup but getting loads of "permission denied error messages"

Code:

backup_action () {
          update_timestamp    # function to get real-time
          LOGFILE="$TMP/$ACTION-$TIMESTAMP.log"  # log file named
          touch $LOGFILE #  log file created
          echo "$LOGFILE saved to $TMP"  

            if [[ -d "$BACKUP_FILES_DIRECTORY" ]]; then
             BACKUP_LOCATION=`echo "$BACKUP_FILES_DIRECTORY" | sed s'/\/$//'` # del "/"
             BACKUP_FILE="$BACKUP_LOCATION/backup-$TIMESTAMP.tar.gz"  
              echo "$TIMESTAMP Beginning Backup - Running Backup now:"  >> $LOGFILE
              awk 'NR==20,NR==EOF' "$CONFILE" > "$TMP/backuplist.txt"  # getting file list
                  BACKUP_TARGETS="$TMP/backuplist.txt"  
                 echo -e "\nBacking up these files now:"  >> "$LOGFILE"
                tar -czvf $BACKUP_FILE -T "$BACKUP_TARGETS" > "$LOGOUT" 2> "LOGTMPERR"
               echo "$TIMESTAMP BACKUP file $BACKUP_FILE created" >> $LOGFILE
               echo -e "\nWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW\n" >> $LOGFILE
            fi
}

Getting loads of errors as ROOT and normal?

root@ubuntu:/home/ubuntu# ./assignment1.sh
/home/ubuntu/confile.cfg: line 20: /home/ubuntu/assignment1/important_files/file1: Permission denied
/home/ubuntu/confile.cfg: line 21: /home/ubuntu/assignment1/important_files/file2: Permission denied
/home/ubuntu/confile.cfg: line 22: /home/ubuntu/assignment1/important_files/file3: Permission denied
/home/ubuntu/confile.cfg: line 23: /home/ubuntu/assignment1/important_files/file9: Permission denied
Config file exists and is located in same Directory as: assignment1.sh
touch: cannot touch `/root/assignment1/logs/temp.err': No such file or directory
touch: cannot touch `/root/assignment1/logs/log.err': No such file or directory
touch: cannot touch `/root/assignment1/logs/backup_out': No such file or directory
$IMPORTANT_FILES_DIRECTORY is present

shivaa · 03-10-2013, 12:18 PM

I do not understand why you've made it so complicated... It seems like everything has messed up!

It's not easy to say anything without seeing the whole script. But I doubt, following lines could cause a problem:

Code:

BACKUP_LOCATION=`echo "$BACKUP_FILES_DIRECTORY" | sed s'/\/$//'` # del "/"
OR 
awk 'NR==20,NR==EOF' "$CONFILE" > "$TMP/backuplist.txt"  # getting file list

Also it is better to define all variables at the beginning of the script or function. Below line could be:

Code:

awk 'NR==20,NR==EOF' "$CONFILE" > "$TMP/backuplist.txt"  # getting file list
                  BACKUP_TARGETS="$TMP/backuplist.txt"

Replaced with:

Code:

BACKUP_TARGETS="$TMP/backuplist.txt"
awk 'NR==20,NR==EOF' "$CONFILE" > $BACKUP_TARGETS  # getting file list

So before doing or suggesting anything more, once run your script with set -xv and post the snippet of erorrs.

Code:

#!/bin/bash
set -xv
REST SCRIPT...

jpollard · 03-11-2013, 03:53 AM

As a side note (and this has nothing directly to do with how to fix your script).

I do understand it can be VERY useful to annotate configuration files like the list of "important files".

You CAN use the comments as you have them, but to do so requires you to remove the comments before sending the data to tar. One way to do this is to use stdin to read the list of important files:

Code:

grep -v '^#' list | tar -T - -cf test.tar

The "-T -" causes tar to read the list from stdin instead of from a disk file. Unfortunately, this usage isn't documented (the -T only lists "FILE" as input). Tar usually allows stdin to substitute for parameters that take a file input... but you can use this syntax only once, so another option taking a file for input can't also take it from stdin.

The grep -v '^#' tells grep to drop lines that start with a #...

This could be preferable to creating a temporary file to contain the list. The grep command can be replaced by anything appropriate for filtering out anything not wanted.

You could even use a "hereis" include to allow shell substitution where you want, but that is a bit more inflexible in that to add/remove entries you have to edit the script instead of the initialization data (but it would allow the ~ to be expanded by the shell).

Batistuta_g_2000 · 03-11-2013, 05:11 AM

OK Thanks ran as instructed - so the file reads a config file and the files I want the user to be able to backup are here, but when I run my script it says permission denied?

Code:

# Please list the files or directories to be backed up on the next line, hit enter after each line:
~/assignment1/important_files/file1
++ /home/ubuntu/assignment1/important_files/file1
/home/ubuntu/confile.cfg: line 20: /home/ubuntu/assignment1/important_files/file1: Permission denied
~/assignment1/important_files/file2
++ /home/ubuntu/assignment1/important_files/file2
/home/ubuntu/confile.cfg: line 21: /home/ubuntu/assignment1/important_files/file2: Permission denied
~/assignment1/important_files/file3
++ /home/ubuntu/assignment1/important_files/file3
/home/ubuntu/confile.cfg: line 22: /home/ubuntu/assignment1/important_files/file3: Permission denied
~/assignment1/important_files/file9
++ /home/ubuntu/assignment1/important_files/file9
/home/ubuntu/confile.cfg: line 23: /home/ubuntu/assignment1/important_files/file9: Permission denied

shivaa · 03-11-2013, 08:02 AM

Seems that files are owned by some other user, and you're invoking script with some other username:

So check if you have read/write permissions for specified files, for which you're getting permission denied:

Code:

~$ while read file; do ls -la $file; done < backuplist.txt

Or simply check:

Code:

~$ ls -la /home/ubuntu/confile.cfg /home/ubuntu/assignment1/important_files/file1 /home/ubuntu/assignment1/important_files/file2 ....

Batistuta_g_2000 · 03-11-2013, 08:20 AM

Quote:

Originally Posted by shivaa

Seems that files are owned by some other user, and you're invoking script with some other username:

So check if you have read/write permissions for specified files, for which you're getting permission denied:

Code:

~$ while read file; do ls -la $file; done < backuplist.txt

Or simply check:

Code:

~$ ls -la /home/ubuntu/confile.cfg /home/ubuntu/assignment1/important_files/file1 /home/ubuntu/assignment1/important_files/file2 ....

getting -rw-rw-r-- 1 ubuntu ubuntu for all, but why would I need rw for the file - am I not just reading from all? - will change and give it a go. Thanks

ubuntu@ubuntu:~$ ls -la /home/ubuntu/confile.cfg
-rw-rw-r-- 1 ubuntu ubuntu 1088 2013-03-11 09:52 /home/ubuntu/confile.cfg
ubuntu@ubuntu:~$ ~$ ls -la ~/assignment1/important_files/file9
~$: command not found
ubuntu@ubuntu:~$ ~$ ls -la /home/ubuntu/assignment1/important_files/file9
~$: command not found
ubuntu@ubuntu:~$ ls -la /home/ubuntu/assignment1/important_files/file9
-rw-rw-r-- 1 ubuntu ubuntu 6 2013-03-07 14:22 /home/ubuntu/assignment1/important_files/file9
ubuntu@ubuntu:~$ ls -la /home/ubuntu/assignment1/important_files/file1
-rw-rw-r-- 1 ubuntu ubuntu 77 2013-03-07 14:20 /home/ubuntu/assignment1/important_files/file1
ubuntu@ubuntu:~$ ls -la /home/ubuntu/assignment1/important_files/file2
-rw-rw-r-- 1 ubuntu ubuntu 252 2013-03-07 14:21 /home/ubuntu/assignment1/important_files/file2
ubuntu@ubuntu:~$

shivaa · 03-11-2013, 08:32 AM

What's content of $CONFILE, so we can check what and how is this command doing?:

Code:

awk 'NR==20,NR==EOF' "$CONFILE" > "$TMP/backuplist.txt"  # getting file list

Batistuta_g_2000 · 03-11-2013, 08:38 AM

Quote:

Originally Posted by shivaa

What's content of $CONFILE, so we can check what and how is this command doing?:
[code]awk 'NR==20,NR==EOF' "$CONFILE" > "$TMP/backuplist.txt" # getting file list[code]

The confile.cfg from row 20 to en of file is:

# Please list the files or directories to be backed up on the next line, hit enter after each line:
~/assignment1/important_files/file1
~/assignment1/important_files/file2
~/assignment1/important_files/file3
~/assignment1/important_files/file9

When I run the script with set -xv I get this near the start (when reading from the confile.cfg)
I have changed to chmod 755 for all files in important_files directory - but is the script trying to run them? why "command not found?" - It seems o work but I am not sure if is working properly?

Thanks (set -xv is great help)

Code:



# Please list the files or directories to be backed up on the next line, hit enter after each line:
~/assignment1/important_files/file1
++ /home/ubuntu/assignment1/important_files/file1
/home/ubuntu/assignment1/important_files/file1: line 1: kjsdf: command not found
/home/ubuntu/assignment1/important_files/file1: line 2: asdfj: command not found
/home/ubuntu/assignment1/important_files/file1: line 3: asdfjasdf: command not found
/home/ubuntu/assignment1/important_files/file1: line 4: asdf: command not found
/home/ubuntu/assignment1/important_files/file1: line 5: asdf: command not found
/home/ubuntu/assignment1/important_files/file1: line 6: asdf: command not found
/home/ubuntu/assignment1/important_files/file1: line 7: bdrtwertewgrt: command not found
/home/ubuntu/assignment1/important_files/file1: line 8: gsbvwgtr: command not found
/home/ubuntu/assignment1/important_files/file1: line 9: bwetbtrw: command not found
/home/ubuntu/assignment1/important_files/file1: line 10: bwdfre: command not found
~/assignment1/important_files/file2
++ /home/ubuntu/assignment1/important_files/file2
/home/ubuntu/assignment1/important_files/file2: line 1: rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr: command not found
/home/ubuntu/assignment1/important_files/file2: line 2: rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr: command not found
/home/ubuntu/assignment1/important_files/file2: line 3: rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr: command not found
/home/ubuntu/assignment1/important_files/file2: line 4: rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr: command not found
~/assignment1/important_files/file3
++ /home/ubuntu/assignment1/important_files/file3
/home/ubuntu/assignment1/important_files/file3: line 1: werefdsfgdsfg: command not found
/home/ubuntu/assignment1/important_files/file3: line 2: dsrrgfdsfg: command not found
/home/ubuntu/assignment1/important_files/file3: line 3: dsfgdfsg: command not found
/home/ubuntu/assignment1/important_files/file3: line 4: ewrterwt: command not found
/home/ubuntu/assignment1/important_files/file3: line 5: sdfgsfdbgsdrtgerw: command not found
/home/ubuntu/assignment1/important_files/file3: line 6: sdfbdsfgf: command not found
~/assignment1/important_files/file9
++ /home/ubuntu/assignment1/important_files/file9
/home/ubuntu/assignment1/important_files/file9: line 1: dsfg: command not found

Batistuta_g_2000 · 03-11-2013, 09:12 AM

Running script but not properly - getting all this "output" of file content but is not backing them up properly?

/assignment1.sh
/home/ubuntu/assignment1/important_files/file1: line 1: kjsdf: command not found
/home/ubuntu/assignment1/important_files/file1: line 2: asdfj: command not found
/home/ubuntu/assignment1/important_files/file1: line 3: asdfjasdf: command not found
/home/ubuntu/assignment1/important_files/file1: line 4: asdf: command not found
/home/ubuntu/assignment1/important_files/file1: line 5: asdf: command not found
/home/ubuntu/assignment1/important_files/file1: line 6: asdf: command not found
/home/ubuntu/assignment1/important_files/file1: line 7: bdrtwertewgrt: command not found
/home/ubuntu/assignment1/important_files/file1: line 8: gsbvwgtr: command not found
/home/ubuntu/assignment1/important_files/file1: line 9: bwetbtrw: command not found
/home/ubuntu/assignment1/important_files/file1: line 10: bwdfre: command not found
/home/ubuntu/assignment1/important_files/file2: line 1: rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr: command not found
/home/ubuntu/assignment1/important_files/file2: line 2: rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr: command not found
/home/ubuntu/assignment1/important_files/file2: line 3: rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr: command not found
/home/ubuntu/assignment1/important_files/file2: line 4: rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr: command not found
/home/ubuntu/assignment1/important_files/file3: line 1: werefdsfgdsfg: command not found
/home/ubuntu/assignment1/important_files/file3: line 2: dsrrgfdsfg: command not found

shivaa · 03-11-2013, 09:50 AM

Files and their permissions seem ok. But I don't know from where these command not found errors are coming.

I feel, it would be better if you can post whole script once (if something is confidential, then you can hide that and use dummy entry at that place).

Batistuta_g_2000 · 03-11-2013, 10:01 AM

Quote:

Originally Posted by shivaa

Files and their permissions seem ok. But I don't know from where these command not found errors are coming.

I feel, it would be better if you can post whole script once (if something is confidential, then you can hide that and use dummy entry at that place).

Is there any way to reply privately - this is for a course assignment and the lecturer checks all sites to make sure we have not copied - will be impossible to prove - can I send direct?