[SOLVED] apt-offline fails to install updates to offline computer with ERROR: Failed to unlink /var/lib/apt/lists/partial/

JoLinuxUser · 08-28-2019, 03:28 PM

I've been working to get apt-offline to update an offline ubuntu box running 16.04, but I've ran into some problems.

I believe software is installed correctly on the offline box. I run the following command with no errors:

Code:

sudo apt-offline set /var/tmp/apt-updates.sig

I then take the resulting .sig file to a Windows PC running apt offline and I'm able to pull down the updates.

If I then try to install the updates with the following command, I get ERRORs and the update never happens

I run:

Code:

sudo apt-offline install apt-offline.zip --verbose

I get the following:

Code:

VERBOSE: Namespace(allow_unauthenticated=False, func=<function installer at 0x7f591efb7b18>, install='apt-offline.zip', install_src_path=None, simulate=False, skip_bug_reports=False, verbose=True)
VERBOSE: Cleaning old update data file /var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_xenial-updates_Release.gpg.
VERBOSE: Cleaning old update data file /var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_xenial_Release.
VERBOSE: Cleaning old update data file /var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_xenial_InRelease.
VERBOSE: Cleaning old update data file /var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_xenial-updates_Release.
VERBOSE: Cleaning old update data file /var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_xenial-backports_Release.
VERBOSE: Cleaning old update data file /var/lib/apt/lists/partial/archive.canonical.com_ubuntu_dists_xenial_Release.gpg.
VERBOSE: Cleaning old update data file /var/lib/apt/lists/partial/archive.canonical.com_ubuntu_dists_xenial_Release.
VERBOSE: Cleaning old update data file /var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_xenial-updates_InRelease.
VERBOSE: Cleaning old update data file /var/lib/apt/lists/partial/security.ubuntu.com_ubuntu_dists_xenial-security_InRelease.
VERBOSE: Cleaning old update data file /var/lib/apt/lists/partial/archive.canonical.com_ubuntu_dists_xenial_InRelease.
VERBOSE: Cleaning old update data file /var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_xenial_Release.gpg.
VERBOSE: Cleaning old update data file /var/lib/apt/lists/partial/security.ubuntu.com_ubuntu_dists_xenial-security_Release.gpg.
VERBOSE: Cleaning old update data file /var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_xenial-backports_Release.gpg.
VERBOSE: Cleaning old update data file /var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_xenial-backports_InRelease.
VERBOSE: Cleaning old update data file /var/lib/apt/lists/partial/security.ubuntu.com_ubuntu_dists_xenial-security_Release.
VERBOSE: {}
VERBOSE: Great!!! No bugs found for all the packages that were downloaded.

VERBOSE: File is security.ubuntu.com_ubuntu_dists_xenial-security_Release.gpg, hence 'True'.
VERBOSE: security.ubuntu.com_ubuntu_dists_xenial-security_Release.gpg file synced to /var/lib/apt/lists/partial.
VERBOSE: security.ubuntu.com_ubuntu_dists_xenial-security_Release file synced to /var/lib/apt/lists/partial.
VERBOSE: security.ubuntu.com_ubuntu_dists_xenial-security_InRelease file synced to /var/lib/apt/lists/partial.
ERROR: Failed to unlink /var/lib/apt/lists/partial/security.ubuntu.com_ubuntu_dists_xenial-security_main_binary-amd64_Packagesapt-offline
ERROR: Failed to unlink /var/lib/apt/lists/partial/security.ubuntu.com_ubuntu_dists_xenial-security_main_binary-i386_Packagesapt-offline
ERROR: Failed to unlink /var/lib/apt/lists/partial/security.ubuntu.com_ubuntu_dists_xenial-security_main_i18n_Translation-enapt-offline
ERROR: Failed to unlink /var/lib/apt/lists/partial/security.ubuntu.com_ubuntu_dists_xenial-security_main_dep11_Components-amd64.ymlapt-offline

This continues for every package in my apt-offline.zip file. When I view the contents of /var/lib/apt/lists/partial/ there is nothing there. I'm puzzled at why this is failing since everything seems correct up to this point. Does anyone have any ideas on why this is failing?

business_kid · 08-30-2019, 04:49 AM

Have you tried the usual offenders? Permissions on all directories, disk space, known bug?

JoLinuxUser · 08-30-2019, 08:30 AM

The issue seems related to permissions with /var/lib/apt/lists/partial/. I can't even do an ls on the directory without a sudo first.

I've traced the code down to the part were it is failing. It's failing when it attempts to decompress security.ubuntu.com_ubuntu_dists_xenial-security_main_binary-amd64_Packagesapt-offline.xz into /var/lib/apt/lists/partial.

I'm unable to put any files into this directory, even if I sudo first. looking at the permissions, it seems as _apt is the owner and the only one with Read/write/execute privileges. This box is brand new and there's been no work done on it, so I'm assuming it was configured this way by default. I'm not sure how to set the permissions so that root has access to read/write/execute in this directory. I think it will work if I can do that since I execute the entire apt-offline with a sudo. Any ideas?

ondoho · 08-31-2019, 01:59 AM

^ I think you may have found a bug (esp. since probably not too many people are using apt offline).
You should do your civic duty and report it (or see if it has been reported already and put your +1 there).

I think at some point in the not-too-distant past, apt changed something so it runs as an unprivileged user, not as root.

Be that as it may, show us

Code:

ls -al /var/lib/apt/lists/partial/

please.

- I get permission denied on my stretch box, so sudo might be necessary.
Let's play with it:

Code:

sudo ls -al /var/lib/apt/lists/partial/
sudo ls -al /var/lib/apt/lists/
sudo -u _apt ls -al /var/lib/apt/lists/partial/

JoLinuxUser · 09-09-2019, 06:04 PM

Here is the contents of /var/lib/apt/lists:

Code:

total 1944
drwxr-xr-x 3 root root   4096 Aug 28 04:12 .
drwxr-xr-x 6 root root   4096 Aug 22 05:18 ..
-rw-rw-r-- 1 root root  11474 Aug 29 08:50 archive.canonical.com_ubuntu_dists_xenial_InRelease
-rw-rw-r-- 1 root root  10509 Aug 29 08:50 archive.canonical.com_ubuntu_dists_xenial_Release
-rw-rw-r-- 1 root root    916 Aug 26 18:37 archive.canonical.com_ubuntu_dists_xenial_Release.gpg
-rw-rw-r-- 1 root root 106824 Aug 29 08:50 archive.ubuntu.com_ubuntu_dists_xenial-backports_InRelease
-rw-rw-r-- 1 root root 105859 Aug 29 08:50 archive.ubuntu.com_ubuntu_dists_xenial-backports_Release
-rw-rw-r-- 1 root root    916 Aug 26 18:39 archive.ubuntu.com_ubuntu_dists_xenial-backports_Release.gpg
-rw-rw-r-- 1 root root 246846 Aug 29 08:50 archive.ubuntu.com_ubuntu_dists_xenial_InRelease
-rw-rw-r-- 1 root root 245864 Aug 29 08:50 archive.ubuntu.com_ubuntu_dists_xenial_Release
-rw-rw-r-- 1 root root    933 Aug 26 18:37 archive.ubuntu.com_ubuntu_dists_xenial_Release.gpg
-rw-rw-r-- 1 root root 109196 Aug 29 08:50 archive.ubuntu.com_ubuntu_dists_xenial-updates_InRelease
-rw-rw-r-- 1 root root 108231 Aug 29 08:50 archive.ubuntu.com_ubuntu_dists_xenial-updates_Release
-rw-rw-r-- 1 root root    916 Aug 26 18:38 archive.ubuntu.com_ubuntu_dists_xenial-updates_Release.gpg
-rw-r----- 1 root root      0 Aug 29 08:50 lock
lrwxrwxrwx 1 root root      9 Aug  8 04:50 _Packages -> /Packages
drwx------ 2 _apt root   4096 Aug 29 08:50 partial
-rw-rw-r-- 1 root root 109198 Aug 29 08:50 security.ubuntu.com_ubuntu_dists_xenial-security_InRelease
-rw-rw-r-- 1 root root 108233 Aug 29 08:50 security.ubuntu.com_ubuntu_dists_xenial-security_Release
-rw-rw-r-- 1 root root    916 Aug 26 18:37 security.ubuntu.com_ubuntu_dists_xenial-security_Release.gpg
-rw-r--r-- 1 root root 428669 Aug  8 04:50 Ubuntu%2016.04%20%5fXenial%5f%20-%20Build%20amd64%20LIVE%20Binary%2020160624-10:47_dists_xenial_main_binary-amd64_Packages
-rw-r--r-- 1 root root 340805 Aug  8 04:50 Ubuntu%2016.04%20%5fXenial%5f%20-%20Build%20amd64%20LIVE%20Binary%2020160624-10:47_dists_xenial_main_binary-i386_Packages

Notice _apt owns the directory partial which I believe is the issue. I can list partial if I do:

Code:

sudo ls -la /var/lib/apt/lists/partial

Code:

total 1184
drwx------ 2 _apt root   4096 Aug 29 08:50 .
drwxr-xr-x 3 root root   4096 Aug 28 04:12 ..
-rw-rw-r-- 1 root root  11474 Aug 29 08:50 archive.canonical.com_ubuntu_dists_xenial_InRelease
-rw-rw-r-- 1 root root  10509 Aug 29 08:50 archive.canonical.com_ubuntu_dists_xenial_Release
-rw-rw-r-- 1 root root    916 Aug 26 18:37 archive.canonical.com_ubuntu_dists_xenial_Release.gpg
-rw-rw-r-- 1 root root 106824 Aug 29 08:50 archive.ubuntu.com_ubuntu_dists_xenial-backports_InRelease
-rw-rw-r-- 1 root root 105859 Aug 29 08:50 archive.ubuntu.com_ubuntu_dists_xenial-backports_Release
-rw-rw-r-- 1 root root    916 Aug 26 18:39 archive.ubuntu.com_ubuntu_dists_xenial-backports_Release.gpg
-rw-rw-r-- 1 root root 246846 Aug 29 08:50 archive.ubuntu.com_ubuntu_dists_xenial_InRelease
-rw-rw-r-- 1 root root 245864 Aug 29 08:50 archive.ubuntu.com_ubuntu_dists_xenial_Release
-rw-rw-r-- 1 root root    933 Aug 26 18:37 archive.ubuntu.com_ubuntu_dists_xenial_Release.gpg
-rw-rw-r-- 1 root root 109196 Aug 29 08:50 archive.ubuntu.com_ubuntu_dists_xenial-updates_InRelease
-rw-rw-r-- 1 root root 108231 Aug 29 08:50 archive.ubuntu.com_ubuntu_dists_xenial-updates_Release
-rw-rw-r-- 1 root root    916 Aug 26 18:38 archive.ubuntu.com_ubuntu_dists_xenial-updates_Release.gpg
-rw-rw-r-- 1 root root 109198 Aug 29 08:50 security.ubuntu.com_ubuntu_dists_xenial-security_InRelease
-rw-rw-r-- 1 root root 108233 Aug 29 08:50 security.ubuntu.com_ubuntu_dists_xenial-security_Release
-rw-rw-r-- 1 root root    916 Aug 26 18:37 security.ubuntu.com_ubuntu_dists_xenial-security_Release.gpg

The issue seems to come in when apt-offline tries to uncompress the package into the partial directory. Is there a way to change the ownership here so that apt-offline can write to the partial directory?

ondoho · 09-10-2019, 01:34 AM

Quote:

Originally Posted by JoLinuxUser

Is there a way to change the ownership here

Sure.

Code:

chown -R user:group /var/lib/apt/partial

where you substitute user and group for the user and group you want.
And no, I don't know what user and group that should be.
TBH, I don't even know if that's the issue; your error message from post #1 does not explicitely say "permission denied" anywhere, although that does seem the likely explanation.
I repeat: I have the same permissions & ownerships here, and everything works.
I think you need to turn to the devs of apt-offline.

Or...

Quote:

Originally Posted by ondoho

^ I think you may have found a bug (esp. since probably not too many people are using apt offline).
You should do your civic duty and report it (or see if it has been reported already and put your +1 there).

JoLinuxUser · 09-11-2019, 07:01 PM

I'm posting this here for others who may encounter this. I finally found out what the issue was. It was rather annoying actually, and somewhat difficult to trace down. The culprit turned out to be a missing dependency in python necessary for apt-online to work. Python-lzma is required; however, there is no indication given to the user when it is not installed. This bit of code in AptOfflineCoreLib.py calls a function in APTOfflineLib.py:

Code:

   def magic_check_and_uncompress(self, archive_file=None, filename=None):
                
            if MagicLib is False:
                    log.err("Please ensure libmagic is installed\n")
                    return False

            magicMIME = AptOfflineMagicLib.open(AptOfflineMagicLib.MAGIC_MIME_TYPE)
            magicMIME.load()
            
            retval = False
            if magicMIME.file( archive_file ) == "application/x-bzip2" or magicMIME.file( archive_file ) == "application/gzip" or magicMIME.file(archive_file) == "application/x-xz":
                    temp_filename = os.path.join(self.apt_update_target_path, filename + app_name)
                    filename = os.path.join(self.apt_update_target_path, filename)
                    if magicMIME.file( archive_file ) == "application/x-bzip2":
                        retval = self.decompress_the_file( archive_file, temp_filename, "bzip2" )
                    elif magicMIME.file( archive_file ) == "application/gzip":
                        retval = self.decompress_the_file( archive_file, temp_filename, "gzip" )
                    elif magicMIME.file(archive_file) == "application/x-xz":
            ====>       retval = self.decompress_the_file(archive_file, temp_filename, "xz")
                    else:
                        log.verbose("No filetype match for %s\n" % (filename) )
                        retval = False

It's supposed to return TRUE on success. A closer look at APTOfflineLib.py reveals this:

Code:

modLZMA = True
try:
        import lzma
except ImportError:
        modLZMA = False

which returns FALSE if lzma is not installed (or TRUE if otherwise). Then there's this:

Code:

def decompress_the_file( self, archive_file, target_file, archive_type ):
                '''Extracts all the files from a single condensed archive file'''
                if archive_type == "bzip2" or archive_type == "gzip" or archive_type == "xz":
                        if archive_type == "bzip2":
                                try:
                                        read_from = bz2.BZ2File( archive_file, 'r' )
                                except IOError:
                                        return False
                        elif archive_type == "gzip":
                                try:
                                        read_from = gzip.GzipFile( archive_file, 'r' )
                                except IOError:
                                        return False
                        elif archive_type == "xz":
                                if modLZMA is True:
                                        try:
                                                read_from = lzma.LZMAFile(archive_file, 'rb')
                                        except IOError:
                                                return False //but if we fail, oh well just return FALSE
                                else:
                                        return False
                        else:
                                return False
                                    
                        try:
                                write_to = open ( target_file, 'wb' )
                        except IOError:
                                return False
                        
                        if self.TarGzipBZ2_Uncompress( read_from, write_to ) != True:
                                #INFO: Return False for the stream that failed.
                                return False
                        write_to.close()
                        read_from.close()
                        return True

Which returns FALSE if python-lzma is not installed. Doesn't tell you that it isn't installed, it just fails. I post this so that others may look at this. The lesson learned is to double-check you dependencies!

Firerat · 09-11-2019, 08:11 PM

it is there as a recommends

( this is on Sid )

Code:

apt show apt-offline
Package: apt-offline
<snip>
Depends: python3:any (>= 3.3.2-2~), apt, less, python3-magic
Recommends: python-soappy, debian-archive-keyring, python-lzma, python3-debianbts

You could report it as a dependency to the package maintainer
But I've seen Debian maintainers argue that since --install-recommends is the Debian default those who disable that are on there own (or words to that effect).

ondoho · 09-12-2019, 12:07 AM

Please mark your thread SOLVED.
Others will benefit.

JoLinuxUser · 09-12-2019, 08:11 AM

Firerat you are correct. python-lzma is listed as a dependency, but what makes this difficult is that the offline box is not connected to the internet. I bet this would work great if the box was connected to the internet, but this is the whole point of apt-offline. I was finally able to make .sig files and get the package lists updated, but have yet to try to actually install something new. That is the next (and hopefully final) step. I will mark this thread as solved. Thanks for all those who offered up help.