LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 10-15-2018, 05:21 PM   #1
ChaosInferno
LQ Newbie
 
Registered: Oct 2018
Posts: 6

Rep: Reputation: Disabled
Unhappy Questions on managing storage in Linux


This weekend I ventured into the world of Linux for the first time and am a little confused to say the least. I completed a 13 hour online course and have spent another 10 hours or so reading articles, watching videos and pouring through tutorials. I have learned a lot in a short time and have a solid understanding of a typical Linux directory and mounted storage. I still don't have a solid understanding of the relationship between directories and partitions, or how to effectively apply Links (Soft or Hard). There are plenty of resources that show how to create directories, how to format and mount a new hard drive (HDD). What I don't find is any information on useful ways to apply that knowledge (or what they are typically used for). I'm a very conceptual person and am hoping someone can help me to understand the fundamentals with file storage on Linux; NOT just the how but the what and the why.

**NOTE: One of my pet-peeves is unclear or vague responses. Please be very clear in your responses to this thread. Please state the question or questions you are addressing and provide references where possible. This is not just for me but in anticipation for future beginners who are having the same issues/questions as me. Please throw caution aside and use as much detail as possible. If there is another thread that explains one aspect of this please post a link to that thread.

My Application/Goal: I had an existing desktop running Windows 10. I wanted to continue to use Windows 10 while simultaneously running a CentOS server (this ruled out a dual boot). To accomplish this I decided to use VirtualBox to start a Virtual machine (VM) running CentOS 7. In my desktop I have a 128GB Solid State Drive (SSD) (that I use as a boot drive for Windows 10), one 750GB Hard Drive (HDD), and three 1TB HDDs. My goal was to install CentOS on the 128GB SSD and then to configure the three 1TB HDDs into a RAID 5 drive.

Now I have a VM running CentOS 7 off of an SSD with three formatted HDDs in a RAID 5 configuration that is mounted. As I started to install software (e.g. Apache, MySQL, PHP, PhpMyAdmin, etc.) I started to realize the files were being saved wherever they wanted to go (I assume this is for optimization purposes). I haven't been able to find any information so far on installing software to a specific directory. I'm also noticing that once software is installed it automatically saves files into that same directory.

Now for my first big question (I know these questions are vague):
  • How do I utilize my RAID 5 drive if everything installs and saves files to default locations?
  • How do I manage where software installs and where software saves files?
Every resource I have watched/read on directories and mounting shows a user how to format a drive and then mount the drive to a directory. In every example only one drive (partition) is ever mounted to one directory. In my configuration I started out with one drive (my SSD) and then added a RAID 5 drive. Before adding the RAID 5 drive everything must have been mounted to the SSD so I assume you can mount several directories to the same partition.
  • How do I mount several directories to my RAID 5 drive? Do I simply copy the files to my RAID 5 partition, umount several directories from their current partition, and then mount them all to my RAID 5 partition? How would I then update fstab (separate lines, each listing the same device/UUID, but with a different mount point/directory listed)?
  • What happens if at a later date one of my directories (for example /var/) becomes full and there is no more room on the current SSD or HDD to extend it, how do I move that directory from the existing partition to a new, larger one?
I learned about soft links and hard links but other than describing them as analogous to shortcuts in a Windows OS, none of the resources I have come across give practical applications for either.
  • What are some common uses for links in Linux?
  • Say I have files in a directory stored on my SSD but I wanted to move those files to my RAID 5 (for security). Can I then somehow use links to link those files back to the original directory? For example, say I wanted to move the entire /var/log/ directory. Can I create a /log/ directory on my RAID 5 Drive (copy all of the files from /var/log/ to /raid5/log/ and delete /var/log/) and then create a hard link in /var/ to /raid5/log/? One big concern with moving or linking files/directories is breaking references to software than need specific files to run.
 
Old 10-16-2018, 07:36 AM   #2
rtmistler
Moderator
 
Registered: Mar 2011
Location: USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 9,883
Blog Entries: 13

Rep: Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930
Welcome to LQ
Quote:
Originally Posted by ChaosInferno View Post
This weekend I ventured into the world of Linux for the first time
Quote:
Originally Posted by ChaosInferno View Post
One of my pet-peeves is unclear or vague responses.
Quote:
Originally Posted by ChaosInferno View Post
Now for my first big question (I know these questions are vague)
Please understand that you wrote a book here asking about some very advanced topics.

Therefore please do not be impatient with people who write brief responses, or very lengthy responses. You've asked a lot of questions, some pretty advanced ones, and it is difficult to tell whether or not you've asked things which are far above your current knowledge level, but many members may conclude this due to you citing that you've just started with Linux last weekend. If you're the type who picks up things very rapidly, then this is great, please try to convey that clearly to people by being organized, concise, and clear with what you've done and what your questions are. Not saying you haven't done this, just trying to address your notation there about your pet peeves.

You've booted CentOS, therefore for starters, you have installed Linux.

As to some of your questions:
Quote:
Originally Posted by ChaosInferno View Post
How do I utilize my RAID 5 drive if everything installs and saves files to default locations?
The RAID drive should be for your data and not for your installed programs. Meanwhile the default locations are typically /bin; /sbin; /usr/bin; and /usr/sbin, but /opt; /tmp; /sys; and /var play a factor in the operation of your programs. I'll amend that to say that you can also use the raid drive to store system backups. Meaning I would not "run" from that drive, but I'd perform backups of the system files and programs, and store those where I could restore them. These are different from the data files.
Quote:
Originally Posted by ChaosInferno View Post
How do I manage where software installs and where software saves files?
I noted in my last answer where software installs executable files. Software also may use other system directories while it operates. Once again, this is not your data. Next, where is saves files should be controlled by configuration of the installed software, or if/when you go to save something, you should be given the opportunity to choose where you place your saved result. The default location would be on the /home partition, located in the user sub-directory who is running a given application, as well as some sub-directories further down in their structure, like Documents, or .config, or some other name. Software generally follows some typical practices, but also it can do whatever it has been coded to do. Assuming also that you're familiar with hidden directories? As in the example I cited with .config?
Quote:
Originally Posted by ChaosInferno View Post
How do I mount several directories to my RAID 5 drive?
You can create sub-directories on your raid drive named similar to the data file system for a typical Linux install, such as /home, and then configure your fstab file to mount that location as your /home. Look for online guides as to how to move your /home directory and configure your fstab file, there are plenty.
Quote:
Originally Posted by ChaosInferno View Post
What happens if at a later date one of my directories (for example /var/) becomes full and there is no more room on the current SSD or HDD to extend it, how do I move that directory from the existing partition to a new, larger one?
You can copy and move it to a larger location and then use changes in your fstab to reconfigure the mount point.
Quote:
Originally Posted by ChaosInferno View Post
What are some common uses for links in Linux?
There are many. A very common one is to have a link to an executable file that has a complex name, where the name of the link is typical and more common. So as opposed to you having to type, vi-1.4.32, you just type vi (I made up that example but that's what I'm talking about in general)
Quote:
Originally Posted by ChaosInferno View Post
Say I have files in a directory stored on my SSD but I wanted to move those files to my RAID 5 (for security). Can I then somehow use links to link those files back to the original directory?
Sure. That may be a lot of work.

Perhaps you ought to build this up slowly. Such as first getting your /home partition on the raid and then figuring out what you need, or don't need to back up for system files.

You also should read up on the Linux directory structure, https://www.howtogeek.com/117435/htg...ure-explained/

The next whole chapter is package management and user management. You seem to be highly focused on setting up this raid drive. All good, but what are your intentions here? To set up a system for you? Or to set up a system for several users? Since you are installing a server is why I'm asking. Or are you learning for future career and/or reference?

If this is a system for you and you want to use it as a home server, media server, or personal machine. Once again, protecting your data is fine, but perhaps you ought to learn how to use, modify, tune, and manage the operating system as well as the data itself. Have you read much on the command line?

Last edited by rtmistler; 10-16-2018 at 07:42 AM.
 
2 members found this post helpful.
Old 10-16-2018, 09:27 PM   #3
ChaosInferno
LQ Newbie
 
Registered: Oct 2018
Posts: 6

Original Poster
Rep: Reputation: Disabled
Hi rtmistler! Thank you so much for your time. I'm very excited to hear from you. I indeed wrote a book and appreciate everyone's time. I had posted this question on several other sites (outside of linuxquestions.com) and was getting some pretty sad responses. For example one of the responses I received was "Yes, that is possible but I don't know why you would want to do that." Looking at that response I had no clue what portion of my many questions the poster was responding to; and I wish they would have elaborated on the "I don't know why you would want to do that." I know this was probably someone just trolling me. My only motivation in my **NOTE was to avoid responses like that. I'm trying to be patient but am really excited to get deeper into Linux!

Quote:
You've asked a lot of questions, some pretty advanced ones, and it is difficult to tell whether or not you've asked things which are far above your current knowledge level, but many members may conclude this due to you citing that you've just started with Linux last weekend. If you're the type who picks up things very rapidly, then this is great
I like to think I'm a fast learner. I'm a very conceptual learner and I know I'm lacking in my understanding of the storage system. I do understand the directories and their organization. I also understand the basics of formatting drives and mounting them to a directory. I was hoping to get a better handle of it before I dove too much deeper into other topics.

Quote:
The RAID drive should be for your data and not for your installed programs
I'll plan to only use the RAID for storing files. Thanks for the input.

Quote:
Assuming also that you're familiar with hidden directories? As in the example I cited with .config?
I did learn about the .filename syntax for hidden directories. That confused my quite initially because I couldn't find a lot of the folders in the GUI I'm using (GNOME). I've mostly worked in the Terminal. Another useful tip is using the tab key in the terminal to auto complete commands, etc.

Quote:
There are many. A very common one is to have a link to an executable file that has a complex name, where the name of the link is typical and more common. So as opposed to you having to type, vi-1.4.32, you just type vi (I made up that example but that's what I'm talking about in general)
That is very clever. Thanks for sharing that trick.

Quote:
The next whole chapter is package management and user management. You seem to be highly focused on setting up this raid drive. All good, but what are your intentions here? To set up a system for you? Or to set up a system for several users? Since you are installing a server is why I'm asking. Or are you learning for future career and/or reference?
My brother and I are starting up an Indie game company and this was the best (lowest cost) solution that I found to store and share large asset files between the two of us. In the near future I would like to learn some HTML and get a website setup as well as an email server (right now we mail forward to gmail and web forward to Google Sites). I already own the desired domain name and am working on getting my CentOS setup to host it. I tried last night be it didn't work some I'm troubleshooting that tonight. It's going to be a fun adventure.

I plan to get GIT and GIT LFS setup very soon as well.

Quote:
You can create sub-directories on your raid drive named similar to the data file system for a typical Linux install, such as /home, and then configure your fstab file to mount that location as your /home. Look for online guides as to how to move your /home directory and configure your fstab file, there are plenty.
This is what I'm having trouble wrapping my head around. I have gone through several online guides but they all show mounting 1 directory to 1 partition. I can't find anything that mounts multiple directories to the same 1 partition.

Say I wanted to move /home to my RAID 5. I formatted my three VDIs and then configured them into RAID 5 naming the partition /dev/md5 (instead of the typical /dev/sdb). I then created directory /raid5 and mounted /raid5 to /dev/md5. This leads me to some conceptual questions. A directory is simply a file that points you to the inodes of other files. When I first create the /raid5 directory that file has to be stored on /dev/sda 1 or 2 since those are the only two partitions in use (lets just pretend it's /dev/sda1). When I mount /dev/md5 to /raid5 does the raid5 directory file move from /dev/sda1 to /dev/md5, or are all of the directories files always stored in the root partition (/dev/sda) and simply start to point to inodes on whatever partition when you mount them to? Not super important questions when it comes to actually using CentOS but just something I'm curious about.

In the guides I've read the next steps in moving the /home directory to my RAID 5 partition would be to create a new directory called /raid5/home. My understanding is that since /dev/md5 is mounted to /raid5 that /raid5/home will save files to /dev/md5. Now I would copy all of the files in /home (files located on /dev/sda1) to /raid5/home (files on /dev/md5); for example rsync -av /home/* /raid5/home. Then delete all of the old content on /home; rm -rf /home/*. Then I would umout /raid5 and then mount /dev/md5 /home (this is where I start to get confused). Conceptually this makes me think all of the directory files are stored on the drive that they point to and when you mount those directories to another partition they get moved to that partition. Otherwise what happens to the /raid5 or /raid5/home directory files when I umount /raid5? They still exist but are no longer pointing to /dev/md5 and could be mounted to some other drive if desired. What if I had other sub-directories on /raid5 for example /raid5/stuff and /raid5/pictures, what would happen to those? The files that /raid5/pictures pointed to are still on /dev/md5. When I umount /raid5 does /raid5/home just become /home, /raid5/stuff become /stuff, and /raid5/pictures become /pictures on /dev/md5? It makes me think that the top level directories like /home and /raid5 are all saved on /dev/sda1 and all of the sub-directories stay on the partition they are created on (continuing to point to files on that partition). I haven't tested this yet but I bet you cannot remove a directory if there are files in it fort his very reason (unless you remove it recursively).

A random conceptual note, when I deleted the content in /home, it technically still points to /dev/sda1 but no longer points to any inodes; the files that /home pointed to are still on /dev/sda1 but /home is no longer keeping track of their location/inodes (and those files will be overwritten the next time any directory decides to use that location on /dev/sda1 to save a file).

So back to me being confused about the umout /raid5 and then mount /dev/md5 /home.The /home folder has no idea what specific files I want it to point to in /dev/md5 so one of two things must be happening, either I'm telling the /home directory that I want it to point to every single file on /dev/md5 (back to one directory and one partition). Or, in the case that /raid5/home becomes /home (on /dev/md5) when I umount it from /raid5, the /home (on /dev/md5) syncs (for lack of a better word) to /home (on /dev/sda1). Which of the two is it (or am I way off and it is neither)?

Now I would need to update the fstab which for one drive to one partition I understand. The syntax I've been taught is as follows:
UUID=<.....> <Mount Point> <filesystem type> defaults 0 2

In the example of moving /home it would be:
UUID=<.....> /home ext4 defaults 0 2

I've also read that you can substitute the partition name (for example /dev/sda1) for the UUID but I don't know if this is actually true (I haven't tried this):
</dev/sda2> <Mount Point> <filesystem type> defaults 0 2

In the case that I wanted to move both /home and /var to my /dev/md5 partition how would I go about doing that? Create another dummy directory like /raid5 and mount /dev/md5 to it? Can I leave /dev/md5 mounted to /home and still mount to /raid5? Do I then create /raid5/var and follow the same steps as before? Once that is completed how would I then setup my fstab? Would I simply add a second line with the same UUID but two different mount points (see example below)?

Pretending my UUID for /dev/md5 is 123456
UUID=123456 /home ext4 defaults 0 2
UUID=123456 /var ext4 defaults 0 2

I did find a different method that uses either symlinks or mount bind. Here is the LINK to that How To.

For the symlinks solution they talk about creating symlinks (I gather this is the same as a soft link) from /home to some other directory say /raid5/home (from our previous example). Then somehow updating fstab to account for that (no example is given). I haven't used symlinks at all yet so I'm not sure how this would be achieved. If this is viable is someone able to please explain how to do this? If a program was to store a file directly in the home directory then I'm guessing I would need to move it to /raid5/home and then create a symlink/soft link of that file back in the /home folder. Looking at my /home directory I see I have a directory for my user so maybe that wouldn't be too hard. It would be a pain in the neck if I had to constantly go in to move files and create links.

For the bind mount solution they show the syntax would be mount --bind /raid5/home /home (for our example above). I'll have to read up on this. Sounds like bind mounts can cause some issues (LINK). If this is a viable solution could someone please elaborate on this as well?

Quote:
Have you read much on the command line?
I have learned a lot of the commands but I don't have them memorized yet. It takes me a while to get anything done right now because I'm constantly having to look up the syntax for a given command. I did learn the --help after a command to list all of the iterations which is really cool. Lots of repetition and I'll eventually start remembering them.

Last edited by ChaosInferno; 10-17-2018 at 07:18 AM.
 
Old 10-17-2018, 04:23 AM   #4
fatmac
LQ Guru
 
Registered: Sep 2011
Location: Upper Hale, Surrey/Hants Border, UK
Distribution: Mainly Devuan, antiX, & Void, with Tiny Core, Fatdog, & BSD thrown in.
Posts: 5,505

Rep: Reputation: Disabled
Hmm.. I'm not going to attempt to answer any of your questions directly, your attitude doesn't sit well with me, but I will point you to a good reference manual for command syntax.

Linux In A Nutshell.
 
1 members found this post helpful.
Old 10-17-2018, 06:19 AM   #5
rtmistler
Moderator
 
Registered: Mar 2011
Location: USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 9,883
Blog Entries: 13

Rep: Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930
@ChaosInferno,

Were they any follow-up questions in your second post in this thread, post #3?
 
1 members found this post helpful.
Old 10-17-2018, 06:40 AM   #6
ChaosInferno
LQ Newbie
 
Registered: Oct 2018
Posts: 6

Original Poster
Rep: Reputation: Disabled
@fatmac

Quote:
your attitude doesn't sit well with me, but I will point you to a good reference manual for command syntax.
Hi fatmac, I appreciate the reference and will check it out. Here is the link to that book on Amazon if anyone else is interested (LINK).

I apologize if something I said offended you as that was not my intent. As to my attitude, something must have been lost in translation; I'm excited to learn more about CentOS and have tried to be patient and polite. I have thanked everyone for their time and patience for responding to my huge post. I re-iterate that I'm very ignorant when it comes to understanding the relationship between directories and partitions in CentOS/Linux. I'm very new at this (only having been doing this since the weekend) and coming from a lifetime with Windows, this concept seems quite different to me.

I would like to prevent future offense. Where did I go wrong? If you are kind enough to elaborate I'll update my original post to clarify both my attitude and motive. I'll be the first to admit that common sense some time evades me. Thanks!

Last edited by ChaosInferno; 10-17-2018 at 07:08 AM.
 
Old 10-17-2018, 07:03 AM   #7
ChaosInferno
LQ Newbie
 
Registered: Oct 2018
Posts: 6

Original Poster
Rep: Reputation: Disabled
@rtmistler

Hi rtmistler, thanks so much for checking back in on me. Again I appreciate all of the time you are taking to help me with my questions.

In a nutshell (pun on fatmac's link ), I followed your input to look for online guides and was only able to find one that spoke directly to mounting multiple directories to the same partition. Their solution used either symlinks or bind mounts. They didn't show any examples so I'm a bit lost on how exactly that would work. I was hoping someone would be able to give more detailed instruction to point me in the right direction (if those are viable solutions).

I thought maybe if I walked through my thought process you would be able to identify where I'm not understanding correctly. I did so in my first reply while following the steps from an online guide for moving the /home directory.

Quote:
So back to me being confused about the umout /raid5 and then mount /dev/md5 /home.The /home folder has no idea what specific files I want it to point to in /dev/md5 so one of two things must be happening, either I'm telling the /home directory that I want it to point to every single file on /dev/md5 (back to one directory and one partition). Or, in the case that /raid5/home becomes /home (on /dev/md5) when I umount it from /raid5, the /home (on /dev/md5) syncs (for lack of a better word) to /home (on /dev/sda1). Which of the two is it (or am I way off and it is neither)?
To clarify, when I mount /home to my RAID 5 drive am I making the whole RAID 5 partition /home?
If the whole drive is now /home then how would I mount another directory to that same partition?


Quote:
Pretending my UUID for /dev/md5 is 123456
UUID=123456 /home ext4 defaults 0 2
UUID=123456 /var ext4 defaults 0 2
If I am able to mount two directories to the same partition (in the example quoted above /dev/md5 is my RAID 5 drive and I'm trying to mount both /home and /var to the same partition), is the above how I should update fstab?

From the guides I've read I'm confident that I understand how to mount one directory to one partition. I'm still unsure as to how I can mount multiple directories to the same partition and then update fstab.

I went back through my second post and bolded the big questions to help them stand out a bit.

Last edited by ChaosInferno; 10-17-2018 at 07:19 AM.
 
Old 10-17-2018, 08:33 AM   #8
rtmistler
Moderator
 
Registered: Mar 2011
Location: USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 9,883
Blog Entries: 13

Rep: Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930
As far as I know you would need multiple UUIDs to do the multiple mounts.

My belief (because this topic extends beyond my experience) is that you'd need to have multiple partitions, a.k.a. multiple objects, in order to attain the multiple UUIDs, and then thus you could mount multiple file systems to the RAID.

If things like LVM or other things allow you to have the multiple objects on the same partition, I do not know. I've not performed those types of storage manipulations.

Perhaps one of our fellow members who does have greater experience on this subject may chime in if they see this thread.
 
Old 10-18-2018, 08:26 PM   #9
ChaosInferno
LQ Newbie
 
Registered: Oct 2018
Posts: 6

Original Poster
Rep: Reputation: Disabled
To consolidate everything above:

System Background: I am running CentOS 7 in a VM (in VirtualBox) on a host computer running Windows 10. My /dev/sda is on an SSD and then I have three 1TB HDDs /dev/sdb, /dev/sdc, and /dev/sdd configured into a RAID 5 drive called /dev/md5. The /dev/md5 is formatted as ext4 and is mounted to /raid5.

I wanted to bind mount /home to a sub-directory in my RAID 5, /dev/md5.

The following steps work for /home:

mkdir /raid5/home
rsync -av /home/*/raid5 /home
mount --bind /raid5/home /home

nano /etc/fstab
/raid5/home /home none bind 0 0

I then restart CentOS and it boots no problem. I check df -aTh and both /raid5/home and /home show mounted to /dev/md5.

FYI, this does NOT work for /var.

Last edited by ChaosInferno; 10-19-2018 at 05:08 PM.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Slackware Packet managing questions johnpuppa Slackware 4 12-21-2013 10:10 AM
package managing questions for newbie wakeboarder3780 Slackware 9 04-25-2006 07:13 PM
DISCUSSION: Network Attached Storage – An Alternative To Tape Drives In Managing Massive Amounts of Data primearray.com LQ Articles Discussion 0 04-02-2006 04:48 PM
little questions about USB storage devices on linux darksmiley Linux - Hardware 5 02-03-2006 05:00 PM
Some very easy questions about managing Linux ACJunior Linux - General 8 12-27-2003 01:48 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 12:49 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration