Hello.

I need to build a website to download big quantity of information; the downloaded data will be saved in many servers, so i have to add a load balancing system.

How can i start?

What is the easyest software to do something like this?

Any help will be welcome.

Kind regards.

Hello and Welcome to LinuxQuestions,

Could you clarify a bit what you mean with building a website to download big quantity of information? Do you mean you are building a website that offers a lot of data that can be downloaded? Or are you doing it the other way around, that is, that your website will download data from other servers and store it on your servers?

Load balancing a website and the traffic isn't that difficult but I'm not sure about the data, how that could be processed.

Kind regards,

Eric

Hello EricTRA and thank you for your answer.

My website will download data from other servers.

Regards.

If you just want to download the data, why would you need load balancing?

I need a load balancing system to decide what server to use whenever it is necessary guard information.

Hi,

So if I understand you correctly you need a distributed data store behind a load balancing program?

Is this what you are looking for?
Distributed Data Store
Sector/Sphere

Kind regards,

Eric

I dont know if "load balancing" or "distributed data store" are the right term.

I have 1 web server and multiple servers to store information. I have to find some way to decide what is the correct data server to use whenever the website needs to download information from other servers.

Hello,

In my opinion if you only have one webserver that will do all the work, then you don't need load balancing. Load balancing is a way of distributing 'the workload' between multiple servers that offer the same service, like a webserver, and that 'hide' behind a virtual IP. For more information about load balancing you can check out heartbeat (Google is your friend).

It seems that you only need the distributed data store to be able to save what's downloaded on multiple servers. I would check out the second link in my previous post (Sector/Sphere). I personally haven't worked with it but it looks like they have a lot of possibilities using scripts. That way you would be able to save data using scripts as filters.

Kind regards,

Eric

My Two cents

You could create a cluster of all nodes - And all nodes in turn could be connected to Single SAN (If you choose - you can use NAS)

Since with a cluster - one of the nodes could be "Master Node" - this node will have channel bonding (preferably 2 bonds - Active Backup/mode 1 - High Availablity) - Should this go off for some reason - Clustered "Download" Service could resume from other node.

Downloads will continue unhampered - and will go to designated RAID/SAN/NAS - whatever.

Asimba: how can i use multiple servers to store X quantity of information with a SAN or a NAS network?

Why a SAN or a NAS network is a better choice than a distributed data store?

Well NAS is nothing but a network storage and computer takes it as a network storage. SAN would be better off. Just for storing data there is no need of cluster. SAN could do your job. And if you are looking for data redundancy, you can get SAN with RAID as well.
I would like to know why are you looking for clustering for just data storage and downloads?

X quantity of information - you could use a daemon/cron job to monitor (ifconfig tx/rx data)

ifconfig will return tx/rx numbers - how you interpret those - you need to work on them.

Secondly - a unified shared storage - is better in terms of maintenance - since you might want to install RAID (Whatever kind you choose) - you wont have to go to individual nodes and take care of RAID/IO scheduling/multi blah blah.

And considering that you have specialized storage stuff like HP EVA - enterprise virtual array - you might be better off with simply plugging in drives as and when needed.

but again - that's lazy me - you might have different opinion.

Im not an expert in networks, so this thread is very difficult to me .

What i still dont understand is if i have to choose between SAN/NAS and a distributed data storage or i need them both?

dunno - but keyword would be to read manuals and never stop at that

NAS - Network attached storage. Its like making a file server and mounting the shares at the client. Its not virtual and client knows that it is using a shared/networked storage.

SAN - Storage area network - Clients will treat it as local storage even though being on network.

SANs could be expensive and will have a dedicated hardware requirement. On the other hand, you can create NAS using some software packages like FreeNAS and installing it on your hardware. A cheaper option for network storage.