[SOLVED] RAID1 clean install questions.

mpyusko · 07-20-2014, 04:43 PM

I am doing a clean install Of Debian64 Wheezy. I have 4 Hard Drives, (2x500 GB and 2x3TB). The 2x3TB's will be a massive (3TB) RAID1 Data array for the ftp/samba/nfs/web server. The 2x500GB's will be RAID1 as well for the OS. The Data array is easy since I already set that one up and I am migrating it to this machine. However the server was originally built with 1 OS drive, but now I want two for redundancy.

The hardware is a FakeRAID controller and it gives me options to create RAID volumes and partitions via BIOS. However I do want the best performance I can get with this setup.

1, Given I will have several partitions on the OS array (<root>, boot, usr, home, var, opt, <swap>,) should I do any configuration in the BIOS to create arrays or should I create them all inside the Debian installer? The documentation I've been reading isn't clear.

Since the Linux Kernel will be handling the arrays, I want to avoid a performance loss due to the BIOS also mirroring. (For specs, see "abacus" in signature)

Edit:

Code:

root@abacus:~# lspci
00:00.0 Host bridge: Intel Corporation E7520 Memory Controller Hub (rev 0c)
00:00.1 Unassigned class [ff00]: Intel Corporation E7525/E7520 Error Reporting Registers (rev 0c)
00:01.0 System peripheral: Intel Corporation E7520 DMA Controller (rev 0c)
00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express Port A (rev 0c)
00:04.0 PCI bridge: Intel Corporation E7525/E7520 PCI Express Port B (rev 0c)
00:05.0 PCI bridge: Intel Corporation E7520 PCI Express Port B1 (rev 0c)
00:06.0 PCI bridge: Intel Corporation E7520 PCI Express Port C (rev 0c)
00:1d.0 USB controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #1 (rev 02)
00:1d.1 USB controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #2 (rev 02)
00:1d.2 USB controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #3 (rev 02)
00:1d.7 USB controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c2)
00:1f.0 ISA bridge: Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC Interface Bridge (rev 02)
00:1f.1 IDE interface: Intel Corporation 82801EB/ER (ICH5/ICH5R) IDE Controller (rev 02)
00:1f.2 RAID bus controller: Intel Corporation 82801ER (ICH5R) SATA Controller (rev 02)
00:1f.3 SMBus: Intel Corporation 82801EB/ER (ICH5/ICH5R) SMBus Controller (rev 02)
01:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge A (rev 09)
01:00.1 PIC: Intel Corporation 6700/6702PXH I/OxAPIC Interrupt Controller A (rev 09)
01:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge B (rev 09)
01:00.3 PIC: Intel Corporation 6700PXH I/OxAPIC Interrupt Controller B (rev 09)
03:01.0 RAID bus controller: Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] Serial ATA Controller (rev 02)
05:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8050 PCI-E ASF Gigabit Ethernet Controller (rev 18)
06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series]
06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series]
07:04.0 Ethernet controller: Intel Corporation 82541GI Gigabit Ethernet Controller (rev 05)
root@abacus:~#

Guttorm · 07-21-2014, 06:34 AM

Hi

I've set up many similar boxes, and some experiences:

- Avoid fake raid. It's not faster than md raid. And when the mainboard dies some years in the future, you can get problems. It's not always easy to find a similar controller. With Raid 1 it's often possible to connect a single drive to a regular SATA port and read it, but not always. Also, you'll be using a driver of some kind, and I think it's more likely to have bug than the well tested kernel's software raid.

- When you read guides, they usually tell you to set up lots of paritions for /usr /tmp /var /home etc. I don't see the point in that. There is a point about the system should not crash if a partition goes full. So if you have one area you are sharing with samba, NFS etc, make that a big separate partioon. The only crital partion, that should never go full is /var. So put /var on a separate partion, or at least separate from the data partion you share.

- If you really want flexibility, set up LVM. But it's more complex, and not neccessary. If you set up many small partition, LVM helps when you need to change sizes.

- Also, 500Gb is a lot for the OS. I would consider making an OS partition, one huge for the sharing, and take out the two 500Gb drives. In a few years, the data drives will proably get full. It would then be easier to have 2 empty disk slots. You can then add two 5 or 6 Tb disks instead. (Max now is 4.5 Tb, but it always increases over the years). In my experience disks tend to be new or full.

- Either leave a few hundred Mb free in the beginning of the disk or make a U(EFI) or FAT partion there. Some newer mainboard use UEFI and want a small partition in the beginning of the disk. Maybe you don't need it now, but maybe you will in a few years time - when you have to replace the mainboard.

- Dont map the disks as sd# in fstab. Use UUID. The installer will set it up that way.

- After installation, run grub-install sd# on all disks in the server. When disks die, or the motherboard dies, it should not matter what order you put the disks in.

And good luck!

mpyusko · 07-21-2014, 08:29 AM

The 3TB's are an upgrade after the previous 1TB's got full. The 500's are overkill, yes. But smaller would have nearly the same cost per drive. I usually setup:

/ 10 GB
/boot 250 MB
/home 40 GB
/opt 10 GB
/use 15 GB
/var 10 GB

The rest is left as flex space for backups and such.

So if I understand you correctly, I should do nothing in BIOS. I should manage the entire configuration from Linux. I'm going to avoid LVM because of its complexity when drives fail.

Guttorm · 07-21-2014, 08:57 AM

Yes, don't set up anything in the BIOS and the OS will find them as separate disks. Then set up software raid with the installer.

I've partition like that before, and often I've been regretting it later. For example /var is too small and /opt is not used at all. But if you have lots of free space you can always fix it later. I don't really see the point of having all those separate partions, except for handeling full disks. So I typically just set up / with 30-40Gb and the rest for /home. Then /home is shared. It gets full all the time, but it doesn't really matter.

And swap should not be needed on a file server, when you have a few Gb RAM. If it needs swap, something is wrong. Having lots of swap does not help at all.

The point removing the 500Gb disks is that I think I would prefer to have two slots empty instead of those disks. You said your previous 1Tb disks got full. When the 3Tb disks get full, wouldn't it be easier to just add 2 more 5Tb disks instead of replacing them or mess with the OS? But if there are lots of empty HD slots and SATA ports in the computer, I guess it doesn't matter.

mpyusko · 07-21-2014, 01:09 PM

Ok, I got it. Thanks.

When I compile from source, often times it installs under /opt. My BOINC projects are also under opt, along with a few other programs. The server usually sits serving and crunching in my basement but periodically I will load up X and pull up something in freecad, or facebook when I'm down there working in the shop. It's self contained in a dust-free, filtered air custom desk. It only has 4GB DDR2. Not huge amounts, but it works.

The primary use is safely storing/serving our photos, videos, documents, and music. But everyone's dropbox is also synced to it which saves overhead. I install dropbox for each user on the server and it handles all the syncing. The users just map to their folders via samba/nfs and I don't need to install dropbox on any of their desktops. This is especially useful for users (like me) who are often dual-booting or using Virtualbox. Installing Dropbox under each OS would mean I could have 3 copies (or more) on a single machine for my Dropbox alone (1 under native win7, 1 under Linux, and 1 under Virtualbox Win7). I also use it for designing, and building websites so I like to leave space for flexibility.