[SOLVED] File is larger when being copied, than what it actually is

pan64 · 01-26-2022, 12:46 AM

probably the target filesystem does not support sparse files. As it was mentioned you might need to convert that qcow2 image to be really less that 20GB before copying (without sparse options).
(it looks like its virtual size is 3 TB, but can be stored in 17 GB).

boughtonp · 01-26-2022, 08:01 AM

Quote:

Originally Posted by Red Squirrel

So what is going on exactly though that it's doing this?
...
This is what I'm trying and it's having trouble though, am I doing something wrong?

Unfortunately I don't know enough about it to answer those - probably worth asking on a Proxmox or QEMU specific forum.

Red Squirrel · 01-26-2022, 12:36 PM

Quote:

Originally Posted by pan64

probably the target filesystem does not support sparse files. As it was mentioned you might need to convert that qcow2 image to be really less that 20GB before copying (without sparse options).
(it looks like its virtual size is 3 TB, but can be stored in 17 GB).

Tried to convert but it seems to not be accepting anything I input for the size. I'm just copying it "in place".

Quote:

root@server04:100# qemu-img convert -f qcow2 -O raw vm-100-disk-0.qcow2 vm-100-disk-0.img -S 18432M
qemu-img: Invalid buffer size for sparse output specified. Valid sizes are multiples of 512 up to 16777216. Select 0 to disable sparse detection (fully allocates output).

But think at this point I'm just going to cut my losses and reformat and restart. Though I will need to figure out a way to stop it from doing this because this is removing my ability of backing up VM files.

pan64 · 01-26-2022, 01:49 PM

you need to convert it without -S, that will create a sparse file. And probably you need to resize it because its virtual size is still 3TB (and you need to shrink it).

Red Squirrel · 01-26-2022, 02:54 PM

I tried without the S option but it just created a file that is acting the same way.

Now I have 2 files that both state they are 3TB even though the entire partition is not even that big. I'm so confused. This is the weirdest thing.

Code:

root@server04:100# df -h
Filesystem           Size  Used Avail Use% Mounted on
udev                  16G     0   16G   0% /dev
tmpfs                3.2G  8.9M  3.1G   1% /run
/dev/md2              20G  3.5G   17G  18% /
tmpfs                 16G   43M   16G   1% /dev/shm
tmpfs                5.0M     0  5.0M   0% /run/lock
tmpfs                 16G     0   16G   0% /sys/fs/cgroup
/dev/sdb1            511M  160K  511M   1% /boot/efi
/dev/mapper/vg-data  3.6T   57G  3.4T   2% /var/lib/vz
/dev/fuse             30M   16K   30M   1% /etc/pve
tmpfs                3.2G     0  3.2G   0% /run/user/1001
root@server04:100# 
root@server04:100# 
root@server04:100# 
root@server04:100# 
root@server04:100# 
root@server04:100# du -h
55G	.
root@server04:100# 
root@server04:100# 
root@server04:100# dir
total 56782316
       0 -rw-r--r-- 1 root root             0 Jan 24 21:45 test
23852408 -rw-r--r-- 1 root root   24424858624 Jan 26 00:54 test.bin
16068288 -rw-r--r-- 1 root root 3221225472000 Jan 26 20:46 vm-100-disk-0.img
16861620 -rw-r----- 1 root root 3221717254144 Jan 24 02:40 vm-100-disk-0.qcow2
root@server04:100#

I think I had set the disk to be 3TB within the VM though. Is PVE doing something weird at the kernel level for the thin provisioning instead of handling it at the application level?

rknichols · 01-26-2022, 09:42 PM

You do need the "-S" option with "qemu-img convert", but with a reasonable size and not that ridiculous 18 gigabytes you used. That size is how large an all-zero block must be in order to be left unallocated at the destination. If the output were in qcow2 format, some (small) multiple of the qcow2 cluster size would be appropriate. For output in raw format, the size should be some multiple of the filesystem block size. If you are just doing this for backup, then a 4096 byte size (1 block) gives the maximum space savings. If you were actually going to mount this copy read/write and use it in operation, then you would want a somewhat larger "-S" size to avoid the filesystem becoming highly fragmented in actual use.

Red Squirrel · 01-27-2022, 07:36 PM

I decided to cut my losses and I will just rebuild the server without backing up this strange file, since I probably don't want a file acting that way anyway.

So I guess my next question is, how do I prevent this from happening again? I just want the VM disk file to act like a normal file. So whatever space it's actually using, I want the file to be that size and if I try to copy it, to stay that size. Ex: like a vmdk would. You can set thin provisioning in vmware to say 3TB, but the vmdk file is going to be smaller, and you can copy it or do what you want with it and it will not try to do anything weird. Can I do this by simply using raw format?