LinuxQuestions.org
Help answer threads with 0 replies.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 03-13-2021, 08:38 PM   #1
browny_amiga
Member
 
Registered: Dec 2001
Location: /mnt/UNV/Mlkway/Earth/USA/California/Silicon Valley
Distribution: Kubuntu, Debian Buster Stable, Windoze 7
Posts: 684

Rep: Reputation: 56
Chain HDDs / SSDs together to form one filesystem without LVM?


Hi,

I have an age-old problem where I'm doing backup and I don't have one hard drive that is large enough for the whole backup.
I want to be able to mount several ones at the same time, to form one large filesystem.
I can do that with LVM, but if one of the devices dies, the whole filesystem gets trashed. I need a system where files are written first on one hard drive, then on the next one and in the case if one hard drive is missing, I can still access the files on the existing one.

Is there such a thing and how is that called? Is that what they call JBOD? (after years I still can't figure out what that is supposed to be, it feels like JBOD means nothing at all, just that you have "a bunch of disks")
 
Old 03-13-2021, 08:48 PM   #2
berndbausch
LQ Addict
 
Registered: Nov 2013
Location: Tokyo
Distribution: Mostly Ubuntu and Centos
Posts: 6,316

Rep: Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002Reputation: 2002
You can use MD RAID.

Never mind; I did not read the full requirement. I don't know if there is a type of filesystem that gives you that much control over disk block allocation AND continues running when half of its storage is broken.

Generally, filesystems tend to keep data that belongs to a file together physically. I remember this from the dawns of UNIX and McKusick's filesystem code, which used the concept of "cylinder groups" for that purpose. This behaviour enables you to recover a good deal of your files in the case of a head crash, but it's no guarantee how much you can recover. And a crashed filesystem should definitely not be used anymore.

Last edited by berndbausch; 03-13-2021 at 08:56 PM.
 
Old 03-13-2021, 09:02 PM   #3
browny_amiga
Member
 
Registered: Dec 2001
Location: /mnt/UNV/Mlkway/Earth/USA/California/Silicon Valley
Distribution: Kubuntu, Debian Buster Stable, Windoze 7
Posts: 684

Original Poster
Rep: Reputation: 56
I used MD RAID 5 and love it, but then I need 3 drives, not just 2. PLUS: it is not fully fault tolerant, if I lose more than one drive, it is game over.
MD RAID on Linux is super solid and reliable.

Now I don't want to use a typical file system to span 2 drives, because you are right, no filesystem will tolerate part of it suddenly missing. A better approach is to format both drives as EXT4 and then use something that spans them together as one filesystem, like mounting them in the same mount point and merging it, there might be something like that. I know that some filesystems allow you to mount a RW portion on a read only filesystem, in order to allow modifications, the live distributions do it like that.
I can't remember what is called.
 
Old 03-13-2021, 10:12 PM   #4
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,130

Rep: Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121
That is a stackable f/s, not really what you are looking for. There have been various incantations of them - one being mergerfs which promises close to what you want. Haven't tried it.

As a passing comment, if you are running into such a brick wall, I'd suggest you need to redesign you backup regime for a tad more flexibilty.
 
Old 03-15-2021, 03:04 PM   #5
jefro
Moderator
 
Registered: Mar 2008
Posts: 21,987

Rep: Reputation: 3628Reputation: 3628Reputation: 3628Reputation: 3628Reputation: 3628Reputation: 3628Reputation: 3628Reputation: 3628Reputation: 3628Reputation: 3628Reputation: 3628
You could really mount any thing to a filesystem to make it transparent to the user I suppose.

A similar thing to LVM is ZFS which has many features beyond LVM. Suppose BtrFS can do it maybe.
 
Old 03-16-2021, 09:34 PM   #6
browny_amiga
Member
 
Registered: Dec 2001
Location: /mnt/UNV/Mlkway/Earth/USA/California/Silicon Valley
Distribution: Kubuntu, Debian Buster Stable, Windoze 7
Posts: 684

Original Poster
Rep: Reputation: 56
Angry

Quote:
Originally Posted by syg00 View Post
That is a stackable f/s, not really what you are looking for. There have been various incantations of them - one being mergerfs which promises close to what you want. Haven't tried it.

As a passing comment, if you are running into such a brick wall, I'd suggest you need to redesign you backup regime for a tad more flexibilty.
I'm using rsync for my backups and that is already very flexible, in that it is using a no-format storage format.
I have just had this problem for decades now and I can't believe that I have not found a solution for it, now is the time to fix this, solve this, once and for all.
Thanks for the tip with mergefs, will take a look at it, it might solve the problem.
 
Old 03-16-2021, 10:29 PM   #7
browny_amiga
Member
 
Registered: Dec 2001
Location: /mnt/UNV/Mlkway/Earth/USA/California/Silicon Valley
Distribution: Kubuntu, Debian Buster Stable, Windoze 7
Posts: 684

Original Poster
Rep: Reputation: 56
Yes, mergefs will do the trick.

As they write on their github page, it supports this very useful feature:
Handling of writes to full drives (transparently move file to drive with capacity)

Now it is just a matter of how to make it work, how to configure it.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Do M.2 SSDs have have the writes issues like older sata SSDs? linux2021 Linux - Newbie 5 08-19-2020 03:35 PM
Should I Get SSDs or HDDs for Linux Builds phdam8 Linux - Newbie 4 07-09-2015 12:13 AM
Linux and SSDs- do they work well together? Earl Parker II Linux - Hardware 9 11-27-2009 07:31 AM
iptables good packet chain (instead of bad packet chain) win32sux Linux - Security 6 11-06-2008 06:02 AM
Accessing other HDDs/Booting from other HDDs Namatacka Ubuntu 2 05-07-2006 11:21 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 04:59 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration