LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 10-19-2016, 09:15 PM   #1
genogebot
Member
 
Registered: Jun 2010
Location: Brisbane, Australia
Distribution: Xubuntu 16.04.1 / Linux Mint 18 XFCE / Linux Mint 18 Mate / Ubuntu Server 16.04.1 / Lubuntu 16.04.1
Posts: 146

Rep: Reputation: 19
Question tar extracted paths not retaining original ownership


I'm creating a tar archive (using gnu tar in Xubuntu 16.04) from selected files in various directories using relative paths. Later I need run a script as root that, among other things, extracts the files from the archive and recreates the paths to the files, but retains the original ownership of the archived files and each of the directories in their path.

When I do this, the extracted files have the correct ownership, but the parent directories are owned by root. Is this the way it's supposed to work? I've read the man page and searched the net but I couldn't find any reference to the ownership of extracted paths.

Here's a vastly simplified example creating an archive using just one file:
Code:
tar -czf test.tar.gz .config/xfce4/xfconf/xfce-perchannel-xml/xfce4-keyboard-shortcuts.xml
The file is later extracted by a shell script run as root:
Code:
tar -xf test.tar.gz
The extracted file 'xfce4-keyboard-shortcuts.xml' retains the correct permissions, but every directory in the path '.config/xfce4/xfconf/xfce-perchannel-xml/' is owned by root, not the owner of the original directories.

I know that archiving the entire directory would preserve its ownership, but I don't want to archive every file in each of these directories, so that is not an option.

I also know I could chown each of the directories in the archived file's path, but there are a lot of disparate files and paths in the archive, and it would be tedious to have to parse the directory tree for every archived file, so that isn't really an option either.

Does tar have some method of extracting the files in such an archive while setting the recreated paths to the correct ownership?

Last edited by genogebot; 10-19-2016 at 09:17 PM.
 
Old 10-19-2016, 10:56 PM   #2
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,860
Blog Entries: 3

Rep: Reputation: 3041Reputation: 3041Reputation: 3041Reputation: 3041Reputation: 3041Reputation: 3041Reputation: 3041Reputation: 3041Reputation: 3041Reputation: 3041Reputation: 3041
In the extraction, you'll want to use the -p option to preserve permissions. It's there in the man page along with some related options.
 
Old 10-20-2016, 01:47 AM   #3
genogebot
Member
 
Registered: Jun 2010
Location: Brisbane, Australia
Distribution: Xubuntu 16.04.1 / Linux Mint 18 XFCE / Linux Mint 18 Mate / Ubuntu Server 16.04.1 / Lubuntu 16.04.1
Posts: 146

Original Poster
Rep: Reputation: 19
Quote:
Originally Posted by Turbocapitalist View Post
In the extraction, you'll want to use the -p option to preserve permissions. It's there in the man page along with some related options.
Thanks for the response. Unfortunately that doesn't work. The man page indicates it shouldn't be necessary for root:
Code:
     -p, --preserve-permissions, --same-permissions
           extract information about file permissions (default for superuser)
I tried it anyway, but the result was the same:
Code:
tree -pufia .
.
[drwxr-xr-x root    ]  ./.config
[drwxr-xr-x root    ]  ./.config/xfce4
[drwxr-xr-x root    ]  ./.config/xfce4/xfconf
[drwxr-xr-x root    ]  ./.config/xfce4/xfconf/xfce-perchannel-xml
[-rw-rw-r-- geno    ]  ./.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-keyboard-shortcuts.xml
 
Old 10-20-2016, 03:14 AM   #4
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,860
Blog Entries: 3

Rep: Reputation: 3041Reputation: 3041Reputation: 3041Reputation: 3041Reputation: 3041Reputation: 3041Reputation: 3041Reputation: 3041Reputation: 3041Reputation: 3041Reputation: 3041
If the tree is pre-existing then "tar" won't change the ownership when extracting over the old directories. Move the old ones out of the way

Code:
mv .config not.config
and then try with -p
 
Old 10-20-2016, 09:20 AM   #5
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,567

Rep: Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093
Quote:
Originally Posted by genogebot View Post
Here's a vastly simplified example creating an archive using just one file:
Code:
tar -czf test.tar.gz .config/xfce4/xfconf/xfce-perchannel-xml/xfce4-keyboard-shortcuts.xml
The file is later extracted by a shell script run as root:
Code:
tar -xf test.tar.gz
The extracted file 'xfce4-keyboard-shortcuts.xml' retains the correct permissions, but every directory in the path '.config/xfce4/xfconf/xfce-perchannel-xml/' is owned by root, not the owner of the original directories.
If you look at the output from "tar -tvf test.tar.tgz" you will see that there is just the single entry for the file itself, nothing for the parent directories. For those directories, there is no ownership or permissions information to restore.

Creating a tar archive that includes directories without recursively including those directories' contents is difficult. You have to exclude the contents. Getting that right for multiple directory levels is quite a mess.
 
Old 10-20-2016, 04:43 PM   #6
genogebot
Member
 
Registered: Jun 2010
Location: Brisbane, Australia
Distribution: Xubuntu 16.04.1 / Linux Mint 18 XFCE / Linux Mint 18 Mate / Ubuntu Server 16.04.1 / Lubuntu 16.04.1
Posts: 146

Original Poster
Rep: Reputation: 19
Quote:
Originally Posted by rknichols View Post
Getting that right for multiple directory levels is quite a mess.
Yes, it does seem that I'm asking tar to do more than it is capable of without some outside help.

This archive-creation script example seems to do the job, storing each parent directory with its ownership, and excluding any other files:

Code:
#!/bin/sh
f=".config/xfce4/xfconf/xfce-perchannel-xml/xfce4-keyboard-shortcuts.xml"
while [ "$f" != "." ]
do 
  tar rf test.tar --no-recursion "$f"
  f="$(dirname "$f")"
done
gzip test.tar
Extracting the contents of test.tar.gz as root creates the directory structure with the correct ownership.

Unless anyone has a better idea I'll mark this as solved.
 
Old 10-20-2016, 07:21 PM   #7
MadeInGermany
Senior Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 1,815

Rep: Reputation: 816Reputation: 816Reputation: 816Reputation: 816Reputation: 816Reputation: 816Reputation: 816
List the archive contents with their owners/permissions
Code:
tar -tvzf test.tar.gz
If the directory owners look correct then extract as root with
Code:
tar -xzf test.tar.gz
Then the extracted directories must have got the owners and permissions in the archive, regardless if they have existed or not.
Otherwise your tar version is buggy.
 
Old 10-20-2016, 09:30 PM   #8
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,567

Rep: Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093
Quote:
Originally Posted by genogebot View Post
Code:
#!/bin/sh
f=".config/xfce4/xfconf/xfce-perchannel-xml/xfce4-keyboard-shortcuts.xml"
while [ "$f" != "." ]
do 
  tar rf test.tar --no-recursion "$f"
  f="$(dirname "$f")"
done
gzip test.tar
You can do it that way as long as you don't want a compressed archive. Appending to a compressed archive is not possible.
 
Old 10-20-2016, 10:54 PM   #9
genogebot
Member
 
Registered: Jun 2010
Location: Brisbane, Australia
Distribution: Xubuntu 16.04.1 / Linux Mint 18 XFCE / Linux Mint 18 Mate / Ubuntu Server 16.04.1 / Lubuntu 16.04.1
Posts: 146

Original Poster
Rep: Reputation: 19
Hmm, it turned out that my idea doesn't work well as a general solution if there is more than one file or directory to be archived in the same directory tree. Tar will happily append any number of duplicates of the same file or folder in the archive, completely ignoring the fact that the archive already contains them; I wasn't expecting that. I ended up with multiples of many of the parent directories. (Presumably this is something to do with tar's origins as a tape backup program.) Tar just isn't that smart.

The only way I've been able to get this to work the way I want is by writing a script to copy each to-be-archived file to a temporary directory, creating its parent directories at the same time, and then using tar to create a gzipped archive of the contents of that temp directory in one pass, then deleting the temp directory. It's a much more convoluted solution than I was hoping for, but everything else I tried ran foul of various limitations in the system utilities.

This is the script I came up with to test the solution on a bunch of files and subdirectories in the same directory:
Code:
#!/bin/sh

set -e

z="$PWD/xfce4-config.tar.gz"

# Define a cleanup function
cleanup()
{
    # Remove temporary directory
    [ -d "$b" ] && [ "$b" = "/tmp/$(basename "$b")" ] && rm -rf "$b"
}

# Run the 'cleanup()' function before exiting
trap cleanup EXIT HUP INT TERM QUIT ABRT TSTP

# Backup any existing archives
[ -f "$z" ] && mv "$z" "$z.backup-$(date +"%F_%H-%M-%S")"

# Create a temporary directory to store copies of the files to be archived
b="$(mktemp --directory --quiet)"

# Change current directory to location of files to be archived
cd "$HOME"

# Relative-path list of files to be archived
LIST=".bashrc
.config/dconf
.config/geany
.config/Thunar
.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-keyboard-shortcuts.xml
.config/xfce4/xfconf/xfce-perchannel-xml/thunar.xml
.config/xfce4/xfconf/xfce-perchannel-xml/thunar-volman.xml
.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-desktop.xml
.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-panel.xml
.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-power-manager.xml
.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-session.xml
.config/xfce4/xfconf/xfce-perchannel-xml/xfwm4.xml
.config/xfce4/panel
.config/xfce4/terminal/terminalrc"

# Parse file list
for l in $LIST
do
    # Copy file list item to temp dir, creating parent directories
    cp --force --parents --update --archive --target-directory="$b" "$l"
done

# Change current directory to temp directory
cd "$b"

# Remove anything that isn't a real file or directory
find . -not -type f -and -not -type d -exec rm -f {} \;

# Finally!
tar -czf "$z" .

# Have a nap...
Fortunately, all the work is required by the archive creation process. The extraction part is still trivial, and extracting as root restores the original ownership:
Code:
$sudo tar -xvf xfce4-config.tar.gz -C /tmp/test
Code:
$tree -augifn /tmp/test
/tmp/test
[geno     geno    ]  /tmp/test/.bashrc
[geno     geno    ]  /tmp/test/.config
[geno     geno    ]  /tmp/test/.config/dconf
[geno     geno    ]  /tmp/test/.config/dconf/user
[geno     geno    ]  /tmp/test/.config/geany
[geno     geno    ]  /tmp/test/.config/geany/filedefs
[geno     geno    ]  /tmp/test/.config/geany/filedefs/filetypes.README
[geno     geno    ]  /tmp/test/.config/geany/geany.conf
[geno     geno    ]  /tmp/test/.config/geany/keybindings.conf
[geno     geno    ]  /tmp/test/.config/geany/tags
[geno     geno    ]  /tmp/test/.config/geany/templates
[geno     geno    ]  /tmp/test/.config/geany/templates/files
[geno     geno    ]  /tmp/test/.config/geany/templates/templates.README
[geno     geno    ]  /tmp/test/.config/Thunar
[geno     geno    ]  /tmp/test/.config/Thunar/accels.scm
[geno     geno    ]  /tmp/test/.config/Thunar/renamerrc
[geno     geno    ]  /tmp/test/.config/Thunar/This is a test file name with spaces
[geno     geno    ]  /tmp/test/.config/Thunar/uca.xml
[geno     geno    ]  /tmp/test/.config/xfce4
[geno     geno    ]  /tmp/test/.config/xfce4/panel
[geno     geno    ]  /tmp/test/.config/xfce4/panel/weather-5.rc
[geno     geno    ]  /tmp/test/.config/xfce4/panel/whiskermenu-1.rc
[geno     geno    ]  /tmp/test/.config/xfce4/terminal
[geno     geno    ]  /tmp/test/.config/xfce4/terminal/terminalrc
[geno     geno    ]  /tmp/test/.config/xfce4/xfconf
[geno     geno    ]  /tmp/test/.config/xfce4/xfconf/xfce-perchannel-xml
[geno     geno    ]  /tmp/test/.config/xfce4/xfconf/xfce-perchannel-xml/thunar-volman.xml
[geno     geno    ]  /tmp/test/.config/xfce4/xfconf/xfce-perchannel-xml/thunar.xml
[geno     geno    ]  /tmp/test/.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-desktop.xml
[geno     geno    ]  /tmp/test/.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-keyboard-shortcuts.xml
[geno     geno    ]  /tmp/test/.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-panel.xml
[geno     geno    ]  /tmp/test/.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-power-manager.xml
[geno     geno    ]  /tmp/test/.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-session.xml
[geno     geno    ]  /tmp/test/.config/xfce4/xfconf/xfce-perchannel-xml/xfwm4.xml
 
Old 10-21-2016, 10:50 AM   #10
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,567

Rep: Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093
Quote:
Originally Posted by genogebot View Post
Hmm, it turned out that my idea doesn't work well as a general solution if there is more than one file or directory to be archived in the same directory tree. Tar will happily append any number of duplicates of the same file or folder in the archive, completely ignoring the fact that the archive already contains them; I wasn't expecting that. I ended up with multiples of many of the parent directories. (Presumably this is something to do with tar's origins as a tape backup program.) Tar just isn't that smart.
The tar program has always allowed multiple versions or multiple copies of a file to appear in the archive. You can use the "--occurrence[=NUMBER]" modifier to extract a particular version.
Quote:
Originally Posted by genogebot View Post
Code:
#!/bin/sh
f=".config/xfce4/xfconf/xfce-perchannel-xml/xfce4-keyboard-shortcuts.xml"
while [ "$f" != "." ]
do 
  tar rf test.tar --no-recursion "$f"
  f="$(dirname "$f")"
done
gzip test.tar
It wouldn't be terribly difficult to keep track of which directories had already been stored.
Code:
#!/bin/sh
declare -A Done
for f in "$@"
do
    while [ "$f" != "." ]
    do 
	[ -z "${Done[$f]}" ] && tar rf test.tar --no-recursion "$f"
	Done[$f]=y
	f="$(dirname "$f")"
    done
done
gzip test.tar
Only the first instance of each path will be stored.

Note that this code (like yours) stores the tree in reverse order. Extraction will fail if any directory does not already exist at the destination. It would be better to create a list, in the proper order, of everything that needs to be stored and then pass that list to a single invocation of tar.
Code:
#!/bin/sh
List=$(mktemp) || exit
trap "rm $List" 0
declare -A Done
for f in "$@"
do
    [ -z "$f" ] && continue
    Fpath=
    while :
    do
	Fpath="$Fpath/${f%%/*}"
	if [ -z "${Done[$Fpath]}" ]; then
	    if [ "$Fpath" = / ]; then  # Hack to handle absolute path
		echo "$Fpath"
	    else
		echo "${Fpath#/}"
	    fi
	fi
	Done[$Fpath]=y
	[[ $f =~ / ]] || break
	f="${f#*/}"
    done
done >$List
tar -czf test.tgz --no-recursion -T $List
Processing those path components would be easier in something other than a shell script. Yes, this script breaks if any path component contains an embedded newline character.

Last edited by rknichols; 10-21-2016 at 03:48 PM. Reason: Handle a null argument sanely
 
Old 10-22-2016, 09:26 PM   #11
genogebot
Member
 
Registered: Jun 2010
Location: Brisbane, Australia
Distribution: Xubuntu 16.04.1 / Linux Mint 18 XFCE / Linux Mint 18 Mate / Ubuntu Server 16.04.1 / Lubuntu 16.04.1
Posts: 146

Original Poster
Rep: Reputation: 19
Quote:
Originally Posted by rknichols View Post
Processing those path components would be easier in something other than a shell script.
Yes, it does get a bit awkward in the shell script.

I decided to do it as a Python script instead, using the tarfile library. Not as compact, but much easier to read.

Code:
#!/usr/bin/env python3
'''
Requires Python 3.5 or later
'''
# pragma pylint: disable=wrong-import-position
import sys
if not sys.version_info[:2] >= (3, 5):  # pylint: disable=unneeded-not
    print("This script requires Python 3.5 or later")
    quit()
from os.path import join, expanduser, dirname, exists  # noqa: E402
from os import chdir, getcwd, rename, stat, walk  # noqa: E402
from time import strftime  # noqa: E402
from stat import ST_MODE, S_ISDIR, S_ISREG  # noqa: E402
import tarfile  # noqa: E402


def main():
    """ Create a gzipped tar archive from a list of files and
        directories, such that there are no duplicates in the archive
        and every parent directory in each item's path can be extracted
        with its original ownership.

        This is a work-around for tar's behaviour in archiving a file
        with a path, where only the file is stored. By default, tar does not
        store a file's parent directories, so their ownership is lost and can't
        be restored on extraction. Extracting the contents from the archive as
        root would then result in the parent directories being recreated with
        root set as the owner, not the original owner.

        This workaround resolves that issue by adding all of the file's parent
        directories to the tar archive as empty directories, enabling each 
        directory's ownership to be retained and subsequently restored on
        extraction.
    """
    # Get the absolute path to the archive file in the current directory
    tpath = join(getcwd(), 'xfce4-config.tar.gz')

    # Change the current directory to the user's home directory
    chdir(expanduser("~/"))

    # The list of relative-path files and directories to be archived
    flist = ['.bashrc',
             '.config/dconf',
             '.config/geany',
             '.config/Thunar',
             ('.config/xfce4/xfconf/xfce-perchannel-xml/'
              'xfce4-keyboard-shortcuts.xml'),
             '.config/xfce4/xfconf/xfce-perchannel-xml/thunar.xml',
             '.config/xfce4/xfconf/xfce-perchannel-xml/thunar-volman.xml',
             '.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-desktop.xml',
             '.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-panel.xml',
             ('.config/xfce4/xfconf/xfce-perchannel-xml/'
              'xfce4-power-manager.xml'),
             '.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-session.xml',
             '.config/xfce4/xfconf/xfce-perchannel-xml/xfwm4.xml',
             '.config/xfce4/terminal/terminalrc',
             '.config/xfce4/panel']

    # Use a 'set' object to automatically purge duplicates
    fset = set()

    # Parse the file list and add items to fset
    for fitem in flist:
        # Get fitem's file mode without following symlinks
        mode = stat(fitem, follow_symlinks=False)[ST_MODE]
        # If fitem is a directory, add its contents to fset
        if S_ISDIR(mode):
            for root, dnames, fnames in walk(fitem):
                for dname in dnames:
                    fset.add(join(root, dname))
                for fname in fnames:
                    fset.add(join(root, fname))
        # Add fitem and any (relative-path) parent directories to fset
        while True:
            if not fitem:
                break
            fset.add(fitem)
            fitem = dirname(fitem)

    # Create a sorted list from fset
    tlist = sorted(fset)

    # Rename any existing file with the archive name
    if exists(tpath):
        rename(tpath, tpath + "." + strftime("%F_%H-%M-%S"))

    # Open a new gzipped tar file in 'exclusive creation' mode (>= Python 3.5)
    archive = tarfile.open(tpath, 'x:gz')

    # Add each item in the sorted list to the tar file,
    # but only if it is a real file or directory, and not a symlink
    for titem in tlist:
        mode = stat(titem, follow_symlinks=False)[ST_MODE]
        if S_ISREG(mode) or S_ISDIR(mode):
            archive.add(titem, recursive=False)

    # Finalize and close the tar file
    archive.close()

if __name__ == '__main__':
    main()
 
Old 10-22-2016, 10:55 PM   #12
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,567

Rep: Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093Reputation: 2093
Quote:
Originally Posted by genogebot View Post
I decided to do it as a Python script instead, using the tarfile library. Not as compact, but much easier to read.
I'll take your word for it. I've never managed to grok Python. I start working through Dive Into Python, and end up as a lump in the belly of a snake.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] How to Add Privilege to A user but Retaining Ownership ojslayout Linux - Server 6 06-30-2015 05:33 PM
[SOLVED] IPTables: reroute port retaining original ip. Cybrax Linux - Networking 7 03-20-2015 05:43 AM
tar archive - How is a single directory extracted without the parent directories? Siopa Linux - Software 12 01-22-2014 03:49 AM
Installing software from extracted tar.bz2 with no ./configure or ./install ryanam Linux - Newbie 17 08-15-2011 05:53 PM
Copy files retaining ownership/permissions stefaandk Linux - Newbie 4 09-07-2006 08:25 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 12:42 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration