Hi folks,

I have a 70Gb file on an EXT4 volume in ~/folder1 that I want to copy to ~/folder2.

generates this error:

The copy process fails when the destination file hits 32Gb in size (or thereabouts).

I've tried using rsync and rclone, and the same thing happens: the copying process fails at around 32Gb destination file size. The volume has 750Gb free space so I do not think it is a disk space problem. It seems like some internal limit is being hit but I haveno idea where to start looking for this.

The EXT4 volume is LUKS encrypted. I do not know whether this is relevant.

I noticed the problem when I was trying to migrate a VirtualBox .vdi file using VBoxManage. The source disk file is 70Gb and the migration failed repeatedly about half way through the migration process. It now seems like an OS problem rather than anything to do with VirtualBox because I cannot cp successfully. I'm not running out of disk space or inodes.

Can anyone suggest a way forward with this?

OS: Arch Linux x86_64 5.17.5-arch1-1

Thanks

VM disk files are often implemented as sparse files, so the guest OS sees a large disk, without actually occupying that space on the host.

See if cp --sparse=when and/or rsync --sparse helps?

I saw an old post abut using the 'nocache' command before the cp,rsync,dd, ect. but it seemed more about performance loss during large file transfers, but it might help. It may not be installed by default it seems, but it should be in your standard repo.

Great idea ... but it didn't work. I got the same Input/output Error at around 32Gb destination file size.

Thanks for the suggestion. I didn't use nocache but, based on some web searching, I tried dd with the following options that disable caching:

I got the same Input/output error, and the copying failed when the destination file reached about 32Gb in size.

Can you copy the file to a different disk? And then back to /folder2?

I should have mentioned this in the original post. No I can't copy the file to a different disk. It hits the same 32Gb limit and generates the Input/output Error.

Do you have a 32bit OS?

I'm running Arch 64-bit.
OS: Arch Linux x86_64 5.17.5-arch1-1

Just seeing that was already answered in your initial post.
Do you get an I/O error?
Maybe a limit in a low level driver, maybe integrated in the SSD.
Did you try to copy to another SSD type or a non-SSD disk?

You seem to have checked a good amount of things so far, but I'll ask how is this disk connected? Is it an external drive or internal? I've seen externally-connected USB drives do flaky things with large files. And can you try to copy another large file, to see if it works? That file itself may be corrupted, or in use.

All of the analysis so far has been directed toward finding out why the output file fails at 32Gb. Is it possible that the error is a physical problem with the input file at 32Gb? I suggest that you run fsck against the partition containing the input file.

2 members found this post helpful.

I have tried copying the file to another folder on the same internal SSD, to an external SSD and to an external HDD. Copying fails every time with the same error.

Another post in this thread suggests that there may be something wrong with the source file. I will investigate this further.

That's a very good suggestion. Thanks.

I have hit a problem with running fsck though. I've booted from a live Ubuntu USB (I had one laying around) but I cannot see the encrypted partition that I need to scan. df -h (and df -a) only shows the partitions associated with the live USB and not the internal volume that I need to scan. I can see the internal volume from the Ubuntu GUI after boot, and can mount it, but cannot open the luks volume from the command line so cannot run fsck on it.

After booting from the live USB, I can mount the encrypted volume from the GUI, then:

This lists all volumes, including the encrypted volume, shown as /dev/dm-012345678 (example only). I unmount this volume, then run

but this produces a "No such file or directory" error. I'm stuck.

This does seem to be getting more complicated and I'm approaching the outer limits of my competence. Thank you to everyone who is helping me with useful suggestions. Who'd have thought that copying a file could be so difficult!

The comment earlier about sparse files and copying is relevant.

You must be able to see the device containing that file system when booted to the live media, but since it is an encrypted volume may not be able to do anything with it except open and mount it.

If you can open the device and not mount it then you should be able to run fsck on it, but if it requires mounting then fsck is out.

You should be able to make a copy of the virtual device using dd, but I have no clue how that would work on an encrypted device, especially one that is a sparse file.