LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 05-09-2009, 09:56 PM   #1
jeopardyracing
Member
 
Registered: Sep 2003
Posts: 175

Rep: Reputation: 30
scalable network storage solution


I have a small business providing IT support to very small companies (30 people and under). We back up our customer's data to our office using rsync over SSH (scripted, using pre shared keys). The service has proven to be so popular that I need a way to expand my storage periodically without creating new logical volumes on the server each time I do. What would be a good Linux solution for this? I'd like to just have one logical volume that I can expand as needed - either by adding storage to a device or by adding nodes to my LAN. My clients are quite small so the budget is very limited (right now I back up 17 clients to a Mac Mini (of all things) that is attached to 4TB of storage in two logical volumes.

Suggestions involving CENTOS are especially appreciated, as this is by far my favorite distro.
 
Old 05-10-2009, 05:12 AM   #2
fotoguy
Senior Member
 
Registered: Mar 2003
Location: Brisbane Queensland Australia
Distribution: KirraMail Live Email Server
Posts: 1,279

Rep: Reputation: 61
There is a distro called openfiler which is a NAS server, (Network Attached Storage), this would be what you are after. It's free and is easy to install. It allows you to add drives as it is needed, and uses a web based administration tool so you can maintain it from anywhere, not sure what distro it is based on, definitely worth looking at.
 
Old 05-10-2009, 08:58 AM   #3
mpiekarski
LQ Newbie
 
Registered: May 2009
Location: Newark, DE
Distribution: Gentoo,ubuntu,rhel
Posts: 25

Rep: Reputation: 16
Hi,

Your first priority should be redundancy. It is one thing to back stuff up, but being able to restore is what counts. Make sure you have some sort of real hardware raid controller helping you out. You might consider something like a Dell MD3000 or MD1000. You can get them refurbed on outlet.dell.com for the cheap (Relatively speaking). They CAN take SATA drives (Up to 15 of them). Perfect if you get 15 TB Drives and set it up as raid 10 or 5 (For maximum redundancy, Go raid 10).

------------------------------------
Michael Piekarski
Network Engineer
mpiekarski@hostmysite.com
www.hostmysite.com
 
Old 05-11-2009, 09:15 AM   #4
jeopardyracing
Member
 
Registered: Sep 2003
Posts: 175

Original Poster
Rep: Reputation: 30
Thanks for the replies; I think the issue with both suggestions is one of cost. In order for this product to be worthwhile financially, we need to keep the costs quite low. I was hoping for an approach a bit more like OpenAFS. Has anyone implemented open AFS? Could I scale a single logical volume just by adding more nodes to my LAN?
 
Old 05-11-2009, 09:44 AM   #5
farslayer
Guru
 
Registered: Oct 2005
Location: Willoughby, Ohio
Distribution: linuxdebian
Posts: 7,228
Blog Entries: 5

Rep: Reputation: 189Reputation: 189
You might want to look at AoE ATA-over-Ethernet

http://en.wikipedia.org/wiki/Coraid

http://www.coraid.com/
 
Old 05-12-2009, 03:54 PM   #6
fotoguy
Senior Member
 
Registered: Mar 2003
Location: Brisbane Queensland Australia
Distribution: KirraMail Live Email Server
Posts: 1,279

Rep: Reputation: 61
Quote:
Originally Posted by jeopardyracing View Post
Thanks for the replies; I think the issue with both suggestions is one of cost. In order for this product to be worthwhile financially, we need to keep the costs quite low. I was hoping for an approach a bit more like OpenAFS. Has anyone implemented open AFS? Could I scale a single logical volume just by adding more nodes to my LAN?
Ok im a little confused, Openfiler is free to use, so it costs you nothing.

Last edited by fotoguy; 05-13-2009 at 02:55 AM.
 
Old 05-12-2009, 05:24 PM   #7
archangel_617b
Member
 
Registered: Sep 2003
Location: GMT -08:00
Distribution: Ubuntu, RHEL/CentOS, Fedora
Posts: 234

Rep: Reputation: 42
Quote:
Originally Posted by jeopardyracing View Post
Thanks for the replies; I think the issue with both suggestions is one of cost. In order for this product to be worthwhile financially, we need to keep the costs quite low. I was hoping for an approach a bit more like OpenAFS. Has anyone implemented open AFS? Could I scale a single logical volume just by adding more nodes to my LAN?
AFS works on a file system level, so I'm not sure this would do what you want. I haven't used AFS before but it's probably worth looking at.

AoE (like another poster mentioned) provides block-level access to storage devices over the network, e.g. a SAN, which may help you. Or iSCSI. Both provide block-level access to storage over a network such that you could add the devices to an existing logical volume on another system.

AoE sends the data in Ethernet frames so it is faster but possibly less reliable (I think in most uses of AoE, it's used on a dedicated storage LAN so this is less of an issue). iSCSI sends the data over TCP so it is slower but more reliable. Since your customers are backing up over the Internet speed is probably not an issue. iSCSI is more widely used / supported.

And Coraid makes good RAID boxes for AoE, but again, you'd have to consider spending some money.

- Arch
 
Old 05-12-2009, 06:21 PM   #8
farslayer
Guru
 
Registered: Oct 2005
Location: Willoughby, Ohio
Distribution: linuxdebian
Posts: 7,228
Blog Entries: 5

Rep: Reputation: 189Reputation: 189
You have to spend money to make money as they say.. You are offering a service to your customers to backup their critical data, and you are going to charge them for that service and perceived security those off-site backups provide, yet you are trying to bargain basement your solution to the point where you won't spend any cash..

I currently backup ALL my servers locally to a large RAID array and to a remote Array/SAN at my backup solution providers location across the internet. similar to what you are describing. I had to pay $12,000.00 for the license for the Secondary Server backup License to run on their server (it's portable license I own it and can setup a secondary server of my own at a later date). The on-site software License was 20,000.00 and annual maintenance / support is around $6,000.00. on top of that I pay for the amount of storage space used on the off-site server storage array. I'm telling you this so you can get an idea of some of the actual costs that can involved to the customer for this type of solution if they want to do it themselves.

Do you have any idea what Data is worth ? If our company lost all their data we would be out of business.. Backups are not something you skimp on., especially with data.that belongs to someone else.

If you aren't even going to attempt to do this properly, by putting a decent storage solution in place, and I'm not even talking something expensive like a SAN, or even what I have in place, just something that is fault tolerant and not cobbled together with twine, spit and duct tape.. . Compliance (HIPPA, PCI, GLBA, etc.. dictates off-site backups and DR planning.. people MUST do this and they should be aware it's not free. The suggestion for the used DELL PE array and the AoE solution are reasonably priced large volume options which you can grow as needed and you seem to have discounted just for the fact that you would have to actually buy something..

Data is worth far more than you seem to admit, please don't do this if you aren't going to be the least bit serious about it. you may be able to afford to screw around, but I would bet your customers can't.

[/rant]
 
Old 07-13-2009, 09:08 PM   #9
jeopardyracing
Member
 
Registered: Sep 2003
Posts: 175

Original Poster
Rep: Reputation: 30
I'm not looking for bargain basement! The economics just have to work. I've priced this product pretty competitively and with only 17 small customers signed up there is no way it would be profitable for $20,000 upfront and and $6,000 per year! We're talking under 3TB of data here.

As the epilogue, here is what I decided on: I copies all the users on a dedicated Centos VM and used a script to copy all of my rsync keys there. Then I have NFS shares that this VM accesses on my physical Centos servers (which only fill the role of VMWare hosts other than this NFS role). I have a central folder on the VM with symbolic links to the NFS mounts. Now when a drive gets too full I can just rsync a few customers' data to a new larger disk on another system, then simply change the symbolic link to hit the new location. Each night, the customers rsync their data to these locations Perfect! Now I have nice scalable storage without the fear of RAIDs collapsing and losing a bunch of customers' data in on swoop. I for one frankly just don't trust RAIDs any more. This way if a disk dies I can just just re copy the customers' data and start fresh. Since these backups are purely for disaster recovery (there is another onsite backup for routine nightly backups there) everything should be nice and safe.

Just wanted to share in case my experience can help others!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Looking for a Data storage solution ufmale Linux - Server 3 06-15-2008 08:18 PM
Solution for storage purposes Swakoo Linux - General 5 04-25-2008 04:26 PM
Scalable Storage Clustering with redundancy? What to use? humbletech99 Linux - Server 2 07-03-2007 04:04 AM
Scalable, cheap disk storage? (Hypothetical, interested in ideas) tormentum Linux - General 4 06-09-2006 02:34 AM
Trying to make a scalable network with Mandrake 9.1 DNS and DHCP Devinci Linux - Networking 0 03-19-2004 02:18 PM


All times are GMT -5. The time now is 09:36 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration