fUnionfs not working as expected

geekdude · 01-08-2024, 11:41 PM

Seeing as Windows 7 is not supported in steam anymore and windows 10 and 11 are annoying I Decided to try installing Nobara linux and see how many of my Steam games will work on it. On a tip that I found on Discord I decided to mount the NTFS partition I have for steam files read only in a union filesystem with an Ext4 partition to save changes. How this was explained to me to work is that you can make changes to the filesystem and they are saved to the read write partition leaving the read only partition intact.

In practice I am finding it takes root privileges to delete most files even though I have 777 access rights and am set as the group and owner on all files in the partition. If I try to delete a file on the union partition without using sudo it shows the error:

Quote:

rm: cannot remove 'foo.exe': Read-only file system

but I can delete it with root which makes me think its a privileges issue maybe. Also I cannot make directories anywhere on the partition It tells me:

Quote:

mkdir: cannot create directory ‘foo’: No such file or directory

Here are the fstab entries for the partitions:

Code:

UUID=7CF6C72C1E7BCB6C                     /mnt/WinSteam  ntfs3   uid=1000,gid=1001,ro,user,exec,umask=000,nofail  0 0
UUID=c90853dc-54aa-462d-bcb6-e07b6ef5e56c /mnt/LinuxSteam ext4  defaults,rw,exec,noatime,nofail 0 0
funionfs#none                  /mnt/Steam     fuse   dirs=/mnt/LinuxSteam=RW:/mnt/WinSteam=ro,uid=1000,gid=1001,rw,exec,allow_other,nofail 0 0

when I try to run any game from steam that is on the union filesystem it shows as needing an update but cannot update and shows a disk write error when I try probably due to the issues I have listed.

I'm wondering if I don't have the configuration right or if I need to look at another filesystem such as aufs, overlay, or a non fuse based version of union.

IsaacKuo · 01-09-2024, 12:05 AM

I'm not 100% sure how this works, but linux is generally case sensitive and most "normal" stuff is lowercase so RW should be rw.

Ages ago I used unionfs but I've long forgotten most of it.

geekdude · 01-09-2024, 10:10 PM

You know what? you are right. But it didn't help. I have also tried it like so and it hasnt changed:

Code:

funionfs#/mnt/LinuxSteam                  /mnt/Steam     fuse   dirs=/mnt/WinSteam=ro,umask=000,allow_other,rw,exec,nofail 0 0

IsaacKuo · 01-09-2024, 10:43 PM

I'm sorry that I really have forgotten most of what I knew about unionfs, and I don't know how different funiunfs is.

What I DO remember is that it was basically more of a pain than it was worth. Just too fiddly and hard to cajole into doing what I wanted.

So I have a dumb brute force suggestion ... maybe just copy over the data to a simple ext4 partition? How much total space are we talking?

rufwoof · 01-10-2024, 07:51 AM

overlayfs is now the Linux preferred choice

mount your windows and linux sets
/mnt/overlay/W
/mnt/overlay/L

create a changes folder where all changes will be stored
mkdir /mnt/overlay/changes

create a top level view folder /mnt/overlay/top
alongside that create a empty work folder /mnt/overlay/work (must be on the same filesystem as overlayfs uses that to move files around that when on the same filesystem is just a pointer action, rather than being a full dataset move action if on another filesystem)

and then

Code:

mount -t overlay -o lowerdir=/mnt/overlay/W:/mnt/overlay/L,upperdir=/mnt/overlay/changes,workdir=/mnt/overlay/work /mnt/overlay/top

You view that using the combined/layered view via /mnt/overlay/top
Any change are stored in /mnt/overlay/changes which IIRC can include whiteout files such as .wh.somefile ... that signal that somefile is technically a deleted file.

So a matter of prserving and retoring the /mnt/overlay/changes folder contents (tar or whatever) between 'reboots'.

geekdude · 01-10-2024, 08:19 PM

Quote:

Originally Posted by IsaacKuo

I'm sorry that I really have forgotten most of what I knew about unionfs, and I don't know how different funiunfs is.

What I DO remember is that it was basically more of a pain than it was worth. Just too fiddly and hard to cajole into doing what I wanted.

So I have a dumb brute force suggestion ... maybe just copy over the data to a simple ext4 partition? How much total space are we talking?

Seeing as Windows 7 is loosing support for steam I downloaded my whole steam library because the games I have already downloaded should still work in offline mode. Its like 1.5 tb but I got a new 8 tb harddrive for Christmas so copying the whole thing is definitely a possibility. I just don't like giving up because I cant figure something out.

Quote:

overlayfs is now the Linux preferred choice

mount your windows and linux sets
/mnt/overlay/W
/mnt/overlay/L

create a changes folder where all changes will be stored
mkdir /mnt/overlay/changes

create a top level view folder /mnt/overlay/top
alongside that create a empty work folder /mnt/overlay/work (must be on the same filesystem as overlayfs uses that to move files around that when on the same filesystem is just a pointer action, rather than being a full dataset move action if on another filesystem)

and then

Code:
mount -t overlay -o lowerdir=/mnt/overlay/W:/mnt/overlay/L,upperdir=/mnt/overlay/changes,workdir=/mnt/overlay/work /mnt/overlay/top
You view that using the combined/layered view via /mnt/overlay/top
Any change are stored in /mnt/overlay/changes which IIRC can include whiteout files such as .wh.somefile ... that signal that somefile is technically a deleted file.

So a matter of prserving and retoring the /mnt/overlay/changes folder contents (tar or whatever) between 'reboots'.

I will look into overlay. The commandline suggested installing overlayfs-tools when I typed overlay and then was able automatically install them. Its probably a lot faster and less fraught with permissions issues than a fuse filesystem.

IsaacKuo · 01-10-2024, 09:03 PM

I'm probably just stating the obvious, but if it's something you do not expect to be able to download again later then you want at least one more copy (on a different drive, not just a different partition) anyway.

Now for something less obvious - once you have a copy on ext4, you can use hardlink "snapshots" to preserve how things were originally, or with "snapshots" later on. No need for wrangling with UnionFS, and you can have as many "snapshots" as you like.

For example, let's say your "live" directory is "/mnt/LinuxSteam/Live/". You can create a hardlink "snapshot" with:

Code:

cp -vaxl /mnt/LinuxSteam/Live /mnt/LinuxSteam/Snap20240110

This does a recursive copy, but "-l" means that it will replicate files with hardlinks rather than duplicating their contents. This makes the snapshot pretty fast and it takes up almost no drive space.

The "magic" is that when a file is deleted, it only gets removed from the ext4 file system when the last hardlink pointing to it is deleted.

The bad news is that if a file is modified in place - such as a log file that is appended to - then all hardlinked copies will be modified. So it's not a true snapshot. I don't know how Steam files work, so this may or may not be a problem.

But this snapshot technique is particularly useful when combined with rsync, because rsync will never update a file in place. If rsync transfers over an updated log file, it will delete the old log file and copy over the new log file.

Anyway, this is just food for thought. There are also "COW" file systems that allow true snapshots. These "Copy On Write" rather than modify files in place.

geekdude · 01-13-2024, 03:08 PM

update: switched to overlay and its working now but not automatically on boot. However it will mount correctly if I run sudo mount -a after it has booted. This is my fstab entry:

overlay /mnt/Steam/SteamLibrary overlay lowerdir=/mnt/WinSteam/SteamLibrary/,upperdir=/mnt/LinuxSteam/SteamLibrary/,workdir=/mnt/LinuxSteam/OverlayTemp/,x-systemd.requires=/mnt/WinSteam/SteamLibrary/,x-systemd.requires=/mnt/LinuxSteam/SteamLibrary/,nofail 0 2

I also see this error 3 times in dmesg:
[ 7.494790] overlayfs: failed to resolve '/mnt/WinSteam/SteamLibrary/': -2

rufwoof · 01-18-2024, 03:25 AM

Quote:

Originally Posted by geekdude

update: switched to overlay and its working now but not automatically on boot. However it will mount correctly if I run sudo mount -a after it has booted. This is my fstab entry:

overlay /mnt/Steam/SteamLibrary overlay lowerdir=/mnt/WinSteam/SteamLibrary/,upperdir=/mnt/LinuxSteam/SteamLibrary/,workdir=/mnt/LinuxSteam/OverlayTemp/,x-systemd.requires=/mnt/WinSteam/SteamLibrary/,x-systemd.requires=/mnt/LinuxSteam/SteamLibrary/,nofail 0 2

I also see this error 3 times in dmesg:
[ 7.494790] overlayfs: failed to resolve '/mnt/WinSteam/SteamLibrary/': -2

Maybe?

https://askubuntu.com/questions/8217...s-to-etc-fstab

Quote:

if your overlay mount references other non-root disks, there is no deterministic guarantee those will be available when the overlay mount is processed (mounts are forked in parallel). This will cause a mount failure during reboots - and enter into maintenance mode. To avoid this add the noauto to the (4th) options field. And add a post-boot script to mount the overlay when all local disks are up.

geekdude · 01-31-2024, 09:31 PM

I got it to work. This is my fstab entry:

Code:

overlay /mnt/Steam/SteamLibrary   overlay   lowerdir=/mnt/WinSteam/SteamLibrary/,upperdir=/mnt/LinuxSteam/SteamLibrary/,workdir=/mnt/LinuxSteam/OverlayTemp/,nofail,x-systemd.requires-mounts-for=/mnt/WinSteam/ 0 0