[SOLVED] tar extract not preserving symlinks

mfoley · 02-09-2017, 08:14 AM

Before ugrading from 13.37 to 14.2 I tar'ed my whole system as

Code:

$ tar --exclude /dev --one-file-system -cvjf tarfile.tar.bz2 /

After the upgrade, I restored various directories, especially user home directories, e.g.:

Code:

$ tar -C / -xvjpf tarfile.tar.bz2 user/mfoley

The problem is, none of the symbolic links were preserved. Examining the actual tarfile:

Code:

$ tar -tvjf tarfile.tar.bz2 user/mfoley/mail/2016
:
lrwxrwxrwx mfoley/develop         0 2016-04-06 11:53 user/mfoley/mail/2016/turboTax2015 -> ../2015/turboTax2015

But after the restore I have:

Code:

$ ls -l ~/mail/2016/turboTax2015
---------- 1 root   root           0 Feb  9 08:40 turboTax2015

This is just one example. It's all over for any file that was a symlink. I also tried `tar -xvjphf` where -h was recommended here: http://www.golinuxhub.com/2013/12/ho...-with-tar.html. It didn't help.

As you can imagine, this is causing me plenty of headaches. Is there something I can do with tar to fix this? I can re-restore some of the important directories

Turbocapitalist · 02-09-2017, 08:46 AM

Are you sure that using -C is not the problem? It puts you into root / but then you are trying to extract a symlink to the directory above it with the double dot short cut .. and that ought to have some consequences, since it won't exist.

mfoley · 02-09-2017, 09:14 AM

more info ... another bizarreness

if I am root, and restore the user's home directory relatively (i.e. /root/user/mfoley), the symlinks work:

Code:

$ tar -xvjpf /backup/2017-02-08-novatec-13.37.tar.bz2 user/mfoley
ls -l user/mfoley/mail/2016/turboTax2015
lrwxrwxrwx 1 mfoley develop       20 Apr  6  2016 turboTax2015 -> ../2015/turboTax2015

Thinking perhaps tar changed after applying 14.2 updates, I removed the original /user/mfoley and tried again:

Code:

$ tar -C / -xvjpf /backup/2017-02-08-novatec-13.37.tar.bz2 user/mfoley/mail/2015
$ ls -l /user/mfoley/mail/2015/turboTax2015
---------- 1 root   root           0 Feb  9 08:40 turboTax2015

Nope, symlink still bad. Can anyone explain this behavior of tar? If not, it looks like my work-around is to restore all directories relatively under root, then move them to the right location.

confused!

mfoley · 02-09-2017, 09:19 AM

Quote:

Originally Posted by Turbocapitalist

Are you sure that using -C is not the problem? It puts you into root / but then you are trying to extract a symlink to the directory above it with the double dot short cut .. and that ought to have some consequences, since it won't exist.

OK, but the double-dotted file is several levels down in /user/mfoley/mail/2016. Why wouldn't it double-dot relative to the directory in which the link was created? I'm not saying you're wrong and my previous test may support your theory, but the behavior doesn't make sense. How would one ever restore a link with a relative reference to ../ ?

rknichols · 02-09-2017, 09:24 AM

That ".." is part of the target of the symlink. A symlink target is just an arbitrary string. It need not refer to anything that exists.

You don't by any chance have "ls" aliased to "ls -L", do you? What do you see with

Code:

\ls -l ~/mail/2016/turboTax2015

GazL · 02-09-2017, 11:17 AM

The tar -h option can do some very, very, odd things. And the guy who wrote that page doesn't know what he's talking about. My advice, don't go anywhere near that option. You certainly don't need it to restore an archive containing symlinks.

55020 · 02-09-2017, 11:45 AM

https://www.gnu.org/software/tar/man.../absolute.html

tl;dr tar + symlinks with .. == bad, but if you really must do that, use the `--absolute-names' (`-P') option when extracting

mfoley · 02-13-2017, 03:57 AM

55020: to quote a few lines from the link you referenced:

Quote:

Symbolic links containing `..' or leading `/' can also cause problems when extracting, so tar normally extracts them last; it may create empty files as placeholders during extraction.

I'm trying to remember if I checked that file before the tar was actually completed. Possibly not. I was restoring that user only and, after it appeared to me that it had extracted all the files, I certainly CTRL-C'd the tar. Maybe if it had run to completion it would have fixed the reference.

Quote:

If you use the `--absolute-names' (`-P') option, tar will do none of these transformations.

To archive or extract files relative to the root directory, specify the `--absolute-names' (`-P') option.

Am I interpreting this correctly to mean that -P on an extract would not wait until last to extract a link properly and not create a placeholder? Would I have had to first save the archive with -P?

Quote:

Normally, tar acts on files relative to the working directory--ignoring superior directory names when archiving, and ignoring leading slashes when extracting.

When you specify `--absolute-names' (`-P'), tar stores file names including all superior directory names, and preserves leading slashes. If you only invoked tar from the root directory you would never need the `--absolute-names' option, but using this option may be more convenient than switching to root.

So, I don't really change to the root directory when creating (maybe I should?), but on extracts I use the -C / switch. Maybe I should also use that switch when creating the archive?

55020 · 02-13-2017, 08:18 AM

Well "after it appeared to me that it had extracted all the files, I certainly CTRL-C'd the tar" is (to put it kindly) material new information that was not previously disclosed.

About all those questions: I don't know. If it matters to you, think it through, try some experiments, and explore the solution space methodically.

rknichols · 02-13-2017, 08:37 AM

Quote:

Originally Posted by 55020

Well "after it appeared to me that it had extracted all the files, I certainly CTRL-C'd the tar" is (to put it kindly) material new information that was not previously disclosed.

Indeed, there are several things that tar defers to the end of processing. Setting permissions on newly created directories is one (the final permissions might not allow creating of files there). It wouldn't surprise me at all that a placeholder might be left instead of a symlink if the extraction were permaturely interrupted.

"Doc, it hurts when I do this." "Don't do that"

mfoley · 02-16-2017, 02:52 PM

I didn't know about tar setting permissions, or doing things with .., etc. after completion. In this case I was scratch-installing a new OS release. The whole tarball backup can take an hour to scan, so I restored specific directories only: /home, /root, /www/tomcat ... and yes, I certainly CTRL-C'd when it appeared the desired directory was done -- or so I thought it was done.

I'm going to assume this was my original permission/symlink problem. I'm too swamped with real work to experiment at the moment, but it makes sense and explains similar issues I ran into some time ago when also doing scratch installs. I've not run into this problem when restoring an entire partition in which case I would let it run to completion.

As I can now do a release upgrade w/o installing from scratch, or clone an OS from a complete partition tarfile, I may never run into this situation again, but good to keep in mind if I need to do a partial restore.

Thanks all.

rknichols · 02-16-2017, 09:09 PM

Quote:

Originally Posted by mfoley

I didn't know about tar setting permissions, or doing things with .., etc. after completion.

Consider a non-root user extracting from a tar archive. If there were a directory in there with permissions that didn't allow writing, and tar set those permissions right away, then extraction of files into that directory would fail. So, tar always creates the directory initially with loose permissions, and defers the setting of the final permissions until the very end. I'm not sure what issues might arise with symlinks. Perhaps a symlink which, later in the archive, was replaced by a file or directory with the same name would be difficult to handle.