Sparse file issue with KVM

david888 · 11-07-2017, 08:22 AM

Hello,

I've installed KVM and and everything is fine. I've been trying to create a VM with Whonix OS. I am having some issues with the file size when I run:

tar -xvf ~/Whonix-Gateway-*.libvirt.xz

and

tar -xvf ~/Whonix-Workstation-*.libvirt.xz

When I run these commands, it eventually errors out and tells me I'm out of space. I am not sure why since I have over 100GB on the partition. So I checked the size of the qcow2 file, and it's 100GB!

I asked this question on the Whonix forum, but they said they believe it is a host operating issue (Trisquel 7) with sparse files.

Any ideas?

(These are the instructions I've been following FYI: https://www.whonix.org/wiki/KVM/Inst...on_Screenshots)

pan64 · 11-08-2017, 08:28 AM

insufficient information, the page you posted cannot be opened.
what will:

Code:

ls -l ~/Whonix-Gateway-*.libvirt.xz
df -h

respond?

rknichols · 11-08-2017, 11:38 AM

Quote:

Originally Posted by david888

Hello,

I've installed KVM and and everything is fine. I've been trying to create a VM with Whonix OS. I am having some issues with the file size when I run:

tar -xvf ~/Whonix-Gateway-*.libvirt.xz

and

tar -xvf ~/Whonix-Workstation-*.libvirt.xz

When I run these commands, it eventually errors out and tells me I'm out of space. I am not sure why since I have over 100GB on the partition. So I checked the size of the qcow2 file, and it's 100GB!

Those files were built strangely. QCOW2 files are not sparse as far as the OS is concerned. Yes, they can have a virtual size larger than the size on disk, but it's not a sparse file.

Here's an analogy. Let's say you have a text file with the following lines:

Code:

At offset 0 there is a string "Hello, world"
At offset 100GB there is a string "End of the world"

That is a representation of a 100GB file, but it's actually just a short text file and is not sparse. QCOW2 represents a virtual file with a non-sparse map that is very much like that.

The QCOW2 file in the Whonix-Gateway archive has a mapping that is just a single 107GB segment with every byte represented literally. Someone just took a 107GB disk image and stuffed it literally into a QCOW2 file format. You'll need at least 107GB of free space to extract it, even though there is only about 4.5GB of non-zero content. On the plus side, that file will never grow any larger no matter how much stuff you store in that virtual disk. It can be converted to a much smaller QCOW2 file by running

Code:

qemu-img convert -S 4096 -O qcow2 Whonix-Gateway-13.0.0.1.4.qcow2 Whonix-Gateway-13.0.0.1.4-minimized.qcow2

but of course you'd have to have the space to store the original 107GB file first.

Here's something to try (untested, but it could work)

Code:

tar --to-stdout -xf Whonix-Gateway-13.0.0.1.4.libvirt.xz Whonix-Gateway-13.0.0.1.4.qcow2 | \
   qemu-img convert -S 4096 -O qcow2 - Whonix-Gateway-13.0.0.1.4.qcow2

Note the backslash at the end of the first line -- that's all one command line, split for readibility.

Nope, forget it, qemu-img won't read from stdin.

voleg · 11-08-2017, 01:16 PM

man tar
Look for "sparse" option.

rknichols · 11-08-2017, 04:39 PM

Quote:

Originally Posted by voleg

man tar
Look for "sparse" option.

Doesn't help. That option only affects adding files to an archive. The QCOL file is not stored sparsely in that archive.

rknichols · 11-08-2017, 05:27 PM

That gives me an idea, though. This will work:

Code:

tar --to-stdout -xf Whonix-Gateway-13.0.0.1.4.libvirt.xz Whonix-Gateway-13.0.0.1.4.qcow2 | \
   dd bs=4k conv=sparse of=temp.qcow2
qemu-img convert -p -S 4096 temp.qcow2 Whonix-Gateway-13.0.0.1.4.qcow2

Again, that first command line is split onto two lines for readability.

That first step yields a bastardized QCOW2 file that qemu-img will see as having a single segment with all the blocks of zeros stored verbatim, but those blocks won't use any space on the disk. The second step converts that to a straightforward QCOW2 file which has only the non-zero blocks mapped. Each of those files uses ~4.3GB of disk space.

Note that in use, that QCOW2 file could grow up to 107GB as you store data into the ext4 filesystem it contains. If you don't anticipate needing that much space (it's using <4.5GB right now), consider using resize2fs to shrink the filesystem and then qemu-img to reduce the virtual QCOW2 size.

voleg · 11-09-2017, 12:12 AM

Quote:

Originally Posted by rknichols

Doesn't help. That option only affects adding files to an archive. The QCOL file is not stored sparsely in that archive.

Of course, you need to use -S when creating archive and when extracting too:

Code:

$ dd if=/dev/zero of=sparse bs=1024k seek=99 count=1 
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0015459 s, 678 MB/s
$ ls -lh sparse 
-rw-rw-r-- 1 voleg voleg 100M Nov  9 08:08 sparse
$ du -h sparse 
1.0M    sparse
$ tar Scf sparse.tar sparse
$ ls -lh sparse.tar  
-rw-rw-r-- 1 voleg voleg 1.1M Nov  9 08:09 sparse.tar
$ rm -f sparse
$ tar Sxf sparse.tar  
$ ls -lh sparse 
-rw-rw-r-- 1 voleg voleg 100M Nov  9 08:08 sparse
$ du -h sparse 
1.0M    sparse

Even without compression on.

rknichols · 11-09-2017, 08:33 AM

Quote:

Originally Posted by voleg

Of course, you need to use -S when creating archive and when extracting too:

Code:

$ dd if=/dev/zero of=sparse bs=1024k seek=99 count=1 
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0015459 s, 678 MB/s
$ ls -lh sparse 
-rw-rw-r-- 1 voleg voleg 100M Nov  9 08:08 sparse
$ du -h sparse 
1.0M    sparse
$ tar Scf sparse.tar sparse
$ ls -lh sparse.tar  
-rw-rw-r-- 1 voleg voleg 1.1M Nov  9 08:09 sparse.tar
$ rm -f sparse
$ tar Sxf sparse.tar  
$ ls -lh sparse 
-rw-rw-r-- 1 voleg voleg 100M Nov  9 08:08 sparse
$ du -h sparse 
1.0M    sparse

Even without compression on.

You'll find that tar behaves exactly the same when extracting without that "S" option. That option affects only creating the archive or adding files to it. It is ignored on extraction.

And anyway, the issue here is with a tar archive that already exists. How it was created is something that cannot be changed now.