Combining multiple directoreis into one

Hegemon · 02-22-2006, 01:14 AM

Does anyone know of any program that can combine several diffrent linux directories into one larger directory?

I have media stored across several drives but would like it to be accessible in one place for browsing on my xbox etc.

Ive been considering using LVM but im worried that if one of the drives fails i loose files across all drives its also alot of setup and adds some complexity.

My other idea is a cron job that runs a script that will link all the files on a regular basis but if there is an actuall software solution aimed more for what I intend i would perfer that.

Mainly it only needs to be read but if there is some basic write support that would be good to although i don't know how it would choose which directory to put the files into.

trickykid · 02-22-2006, 01:32 AM

Are you wanting multiple drives to be spanned across to make it look like one big drive or do you just want to be able to cd into a directory and then also browse your other directories from within that same directory? The first one, use LVM, it's pretty much the only way to accomplish spanned drives to be one large volume. If your looking to do the second way mentioned, just create symlinks.

timmeke · 02-22-2006, 03:34 AM

You can also mount the different drives onto subfolders of the same folder, ie:
/mnt/drive1/
/mnt/drive2/
...
Then all you need is to browse all directories below /mnt on your XBox.

Linking files, ie via a cron job, is not such a good idea. It would be better if you could link directories rather than files.
In this case, you get something like this:
/path/to/media/drive1
/path/to/media/drive2
...
where drive1, drive2, etc are symlinks to the right directories on the different drives.

If you don't want separate directories, but you want all files to appear as if they were in 1 directory,
you could symlink the files, but that would be troublesome. LVM is a more handy solution in this case.

Hegemon · 02-24-2006, 03:26 AM

Basicly if i have the following:
/mnt/storage1/Videos/PerfectallyLegalVideos/video01.avi
/mnt/storage2/Videos/PerfectallyLegalVideos/video02.avi

I want to beable to access all those files from the one directory using some kind of virtual file system.

/mnt/allstoage/Videos/
which had PerfectallyLegalVideos/ and the 2 videos.

Id perfer not to have to combine multiple drives using raid or LVM.

Also i don't think linking the files will work as the xbox media centre combined with samba don't seem to work with the linked files. Seems to have somtrhing to do with how samba treats links.

timmeke · 02-24-2006, 05:53 AM

I think you can configure how Samba treats symbolic links. Samba should be able to dereference them automatically, if I'm not mistaken.
Hard links are not on option, since you're dealing with multiple filesystems.

From what I understand, you want the PerfectallyLegalVideos directories on both filesystems to appear as only 1 directory. LVM or raid solution is the way to go, except if you prefer symlinks.

Remember that your filesystem must be able to know to which storage it needs to address it's read/write calls.
So, you have to either use separate directories (mount points), possibly with symlinks to tie everything together, or use some kind of disk volume management software (like LVM) that acts as a layer between what you see from the filesystem and the actual physical storage. This layer can then pretend that all files are in 1 directory, while storing them on multiple physical disks, for instance.

So sorry, but I don't think you have much choice.

Maybe there are other alternatives to LVM/Raid solutions, but I don't know them.

Note: the correct spelling is "Perfectly" not "Perfectally".

nx5000 · 02-24-2006, 06:26 AM

You can also mimic what LVM does ( I have not tried but will in a few days):
Use the Device Mapper:

dmsetup create fusion
You should find info on google

Hegemon · 02-26-2006, 04:57 AM

Ive decided to link everything togeter using a custom script:

Code:

#!/bin/sh
#dirmerge.sh
#usage dirmerge.sh sourcedir clonedir
realpath=$1
copypath=$2
IFS='
'

#recursivly clone directory structure
find $realpath -mindepth 1 -maxdepth 1 -type d -printf '%f\n' | \
while read I; do
        param1=$copypath/$I
        param2=$realpath/$I
        mkdir -p $param1 2>/dev/null
        $0 $param2 $param1
done

#link all files in current directory to cloned directory
find $realpath -type f -maxdepth 1 -mindepth 1 -printf '%f\n' | \
while read I; do
        param1=$copypath/$I
        param2=$realpath/$I
        ln -s $param2 $param1 2>/dev/null
done

Basicly by using this i should have links of all files including those in the same subdirectories on diffrent drives.

If anyone decides to use it be carefull as its a simple hack and doesn't do any error checking etc, i don't know what it does if you forget one of the pramaters you might end up with a bunch of cloned files in a bad place.

Code:

dirmerge.sh /home/hegemon/stored/Videos /mnt/storage/Videos
dirmerge.sh /mnt/storage1/Videos /mnt/storage/Videos
dirmerge.sh /mnt/storage2/Videos /mnt/storage/Videos

Im still looking at LVM/devicemapper but if any of my harddrives failes that could be very bad, and i don't have enough spare space for parity.

Hegemon · 02-24-2007, 12:37 PM

Just found this old post in my bookmarks, in case anyone searches for this, a much better solution is to use UnionFS which does exactly what I wanted and supports deleting files and writing (to primary drive only though).