LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 02-14-2010, 02:28 AM   #1
ccolumbu
Member
 
Registered: Oct 2009
Posts: 37

Rep: Reputation: 4
Help: How to create a high end NAS/SAN with standard hardware


I have been trying for about 2 years here and there (and failing I might add) to create a "SAN/NAS system" (see my description below) that can export a data partition that can be mounted r/w by multiple cluster nodes. I have tied to use all of these technologies at one point or another in various combinations: drbd, nfs, gfs, ocfs, aoe, iscsi, heartbeat, ldirectord, round robin DNS, pvfs, cman, clvm, and several fencing solutions. I finally "gave up" trying to do it myself, and e-mailed about 10 SAN/NAS appliance re-sellers and only 2 said they could meet my specs. Their price range . . . $10,000 - $80,000. Well that priced me right out of the market.
So back to trying to do it myself. I came across glusterfs, it was super easy to set-up and test and is almost exactly what I want/need. There are only 2 issues.
#1. When one of the servers goes down the client hangs at least for a little while (more testing is needed) to be sure it comes back at all.
#2. The read/write tests I performed came in at 1.6, NFS on all the same machines came in at 11, and a direct test on the data server came in at 111. How do I improve the performance?

###############################################
My gluster set-up:
2 supermicro dual Xeon 3.0 ghz CPUs, 8gb ram, 4 @ 750gb seagate sata HDs, 3 in raid5 with 1 hot spare. (data servers)
1 supermicro dual AMD 2.8 ghz CPUs, 4gb ram, 2 @ 250gb seagate sata HDs in raid 1. (client server)
gluster is set-up with round robin DNS to handle the load balancing of the 2 data servers.
I also tried heartbeat with a virtual IP and ldirectord to redirect the traffic, but when I took down server 1, there was like a 10 second delay before server 2 took over the virtual IP and the client hung.

###############################################
Description of my dream "SAN/NAS system"
Data server:
2 units (appliances/servers) that each have a 4 drives in a raid5 disk set (3 active, 1 hot spare) (they can be active/passive or active/active I don't care)
These two units should mirror each other in in real-time.
If 1 unit fails for any reason the other picks up the load and carries on.
On a failure I want the data to be re-synced automatically when the failed unit comes back on-line.

Data clients:
Each cluster node machine (the clients) in the server farm (CentOS 5.4 OS) will mount 1 or more data partitions provided by the data server(s).
If the active server goes down (multiple HD failure, network issue, power supply, etc) the 2nd server takes over and the client machines never know.
All clients will mount r/w simultaneously, so some type of network file system that supports network file locks is required.

It would be even more ideal if the data servers could be 1-N instead of just 1-2.

To re-cap:
A. 2 SAN/NAS data servers mirroring each other in real time.
B. Auto fail over to the 2nd server if 1st fails (without the clients needing to be restarted, or even being interrupted if possible).
C. Auto re-sync of the data if a failed unit comes back on-line, when the sync is done the unit goes active again (assuming its normal state is active).
D. Multiple machines mounting the same partition in read/write mode (some kind of network file system).
E. Linux CentOS will be used on the cluster nodes.

Can anyone help?
 
Old 02-14-2010, 06:13 AM   #2
hostmaster
Member
 
Registered: Feb 2007
Posts: 55

Rep: Reputation: 17
You can use a third system to export raw devices (GNBD) which can be shared by your two servers but dont know exactly how reliable it is or you can use openfiler (Linux distro) to export file system as iSCSI which can be shared by your servers
 
Old 02-14-2010, 06:34 AM   #3
mesiol
Member
 
Registered: Nov 2008
Location: Lower Saxony, Germany
Distribution: CentOS, RHEL, Solaris 10, AIX, HP-UX
Posts: 731

Rep: Reputation: 137Reputation: 137
Hye,

this is a highly complex configuration you wish to create here. We can discuss this for weeks. Also it is required to know what this cluster should serve. For this purpose there are several different cluster file system available like gfs, ocfs, etc.

Not sure if there is a software available for data mirroring between boxes (your NAS). This kind of mirroring is normally done by vendor specific storage software or as host based mirror with tools like Veritas Storage Foundation. Both are real expensive. I have not found such software for Linux yet, but if someone knows one he should contact me

If the storage part is solved, your cluster should not be the problem.
 
Old 02-14-2010, 11:54 AM   #4
ccolumbu
Member
 
Registered: Oct 2009
Posts: 37

Original Poster
Rep: Reputation: 4
I should have mentioned GNBD

Quote:
Originally Posted by hostmaster View Post
You can use a third system to export raw devices (GNBD) which can be shared by your two servers but dont know exactly how reliable it is or you can use openfiler (Linux distro) to export file system as iSCSI which can be shared by your servers
I should have had GNBD on the list of technologies I have tried.
I was using it for fencing.

As far as I know iscsi is a 1 to 1 technology. Do you know of a way to mount a single iscsi target on multiple machines? What network file system would be used to do the file locking?
 
Old 02-14-2010, 11:58 AM   #5
ccolumbu
Member
 
Registered: Oct 2009
Posts: 37

Original Poster
Rep: Reputation: 4
Quote:
Originally Posted by mesiol View Post
Hye,

this is a highly complex configuration you wish to create here. We can discuss this for weeks. Also it is required to know what this cluster should serve. For this purpose there are several different cluster file system available like gfs, ocfs, etc.

Not sure if there is a software available for data mirroring between boxes (your NAS). This kind of mirroring is normally done by vendor specific storage software or as host based mirror with tools like Veritas Storage Foundation. Both are real expensive. I have not found such software for Linux yet, but if someone knows one he should contact me

If the storage part is solved, your cluster should not be the problem.
Mainly it is serving web content, html, images, perl scripts. It also have asterisk nodes in it so config files, and voice recordings.

There are several mirroring solutions, I mentioned some of them.
Here is a short list (and I am sure it is not complete)
drbd
glusterfs
gfs
ocfs
hdfs
pvfs

However that is only part of the solution, you still need a way to fail-over if one of the servers goes down, and a way to mount the partition multiple times read/write.

I intend to write a detailed how-to if I ever figure this out (of if someone figures it out for me).
 
Old 02-15-2010, 12:56 AM   #6
hostmaster
Member
 
Registered: Feb 2007
Posts: 55

Rep: Reputation: 17
Yes iscsi target can be mounted on multiple nodes but it depends on the file system if shared storage is required. You can try GFS for that. As GFS is a shared storage mechanism you don't need to configure fail-over for that. For NFS you can format the file system as ext3 and configure HA cluster for fail-over scenarios.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
SAN , NAS and IO Scheduling asimba Linux - Enterprise 10 10-26-2009 05:40 AM
Need Help on SAN & NAS srikanthrao_d Linux - Enterprise 4 12-09-2008 10:44 AM
NAS? SAN? I'm lost haiders Linux - Networking 5 09-20-2006 08:25 PM
LXer: Lenovo embeds Linux in high-end and low-end notebooks LXer Syndicated Linux News 0 08-10-2006 11:21 AM
nas vs san Xris718 Linux - General 6 08-25-2005 02:42 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 06:24 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration