I am at the moment running about 3 dedicated servers, but I feel I am not making optimal use of all this hardware.
The servers are generally the same:
- centos 6.4
- 32GB ddr3 ram
- i5 4 core HT @ 3,4Ghz
- 200Mbit uplink
- 2x3TB HDD @ 7200rpm in raid 1

2 servers runs a complete installation of virtualmin. (apache, mysql, dns, php, fcgi, ftp, ...)
One server also runs a RED5 server.
Every server has a zabbix monitoring agent running,
A third server runs the monitoring server and backups. This one has more storage and a slower connection.
Goal of the servers: run several websites of which one has fairly high load.

This is of course ... waisting resources.

I am trying to find an optimal scenario ... Within reasonable costs.
A) Setup dedicated servers for different services
B) Setup virtual machines per service
will this improve the speeds users download the pages?
What kind of resources should I use per service? How could I best calculate this?

Thanks for your insight.

There's certain standards that you should adhere to (like having at least one Name Server in a different network), personally I'd ditch any overhead (like any web-based management panels), caching may help and sure running virtualization may increase efficiency (due to compartmentalizing and assigning resources) but there's simply not enough information to offer suggestions that would suit your situation.

I suggest you start by detailing the amount of web sites, software (names, versions) running in the web stack, what tuning was done already, website popularity, system utilization details and trending, bottleneck pinpointing, etc, etc.

1 members found this post helpful.

Thanks for your comment Unspawn.

At the moment I am running the following:
Server A:
Websites:

• boonex dolphin (about 3 million hits per month)
• Joomla (about 500.000 hits per month

Services:

• memcache
• RED5 media server (for straming & live cam)
• acp
• fcgi
• Zabbix agent
• Mysql
• all the virtualmin services (too many to name and too many to be good)

Server B:
Websites:

• wordpress Multisite (about 1 million hits per month)
• 10 Wordpress standalone install (about 20.000hits per month per site)

(The wordpress sites are subject to increase by about 10 per year)
Services:

• memcache
• acp
• fcgi
• Zabbix agent
• Mysql
• all the virtualmin services (too many to name and too many to be good)

Server C:
Management server containing the zabbix server and backups in a private lan.

I am exploring the possibility of using xenserver for virtualisation.
What would be a good idea to split?
1 memcache server
1 nginx server (instead of now apache)
2 dns server
1 mysql server
1 RED5 media server
Would it be interesting to run all these on 1 server (as the config shows in my first post) or should I use several servers for this?

When you come to think of it 3 million page views per month isn't all that much (given constant and not "bursty" or slashdot-like access) when you view it as hits per hour or second. Any web server should be capable of coping with that without any performance tuning (provided the software in the web stack isn't sub-par). Like I said before using virtualization does make sense because you can compartmentalize and assign resources in a more fine-grained way. However what's still missing to complete the assessment is details wrt database utilization and system utilization in terms of CPU, memory and disk I/O. *While I'm not a web server guru (I usually hang out here) what using Linux taught me is that things conform to rules and that about everything is measurable. I'm not stalling, do understand I need facts to make suggestions that make sense.

1 members found this post helpful.

XenServer is a nice platform with a fairly good and usable interface, however the free version doesn't allow you to do clever things like change the ram or number of processors assigned to a VM "on-the-fly", you have to shut down the VM first.

1 members found this post helpful.

Thanks unSpawn for your comment.
The following are the average statistics I'm getting from my servers:
Server a:
MYSQL: ±650KB/s read, ±5MB/sec write
CPU: ±0,9% usage
RAM: ±10GB usage
Server B uses about half of the above usages.

I do understand that the current setup is certainly sufficient, I'm trying to create a safer and more reliable network without adding more funds.
Any suggestions? fail-over IP's, replicating, load balancing, ...


Thanks TenTenths for your insight into XenServer.
That should be sufficient for now

The funds part you made pretty clear. The problem with something being "safe" or "secure" is that people often don't give too much thought while talking about it and consequently it means often different things to different people. For example for a visitor "safe" may mean data in transit is handled between end points over SSL while for a database admin "safe" may mean having binary log files still guarantees data can be synced. So OK. Let's start by talking about requirements. Generally speaking every running process requires a certain amount of CPU cycles and a minimal amount of memory to run. At the network level every connection (socket) requires memory to receive and process requests. When no more sockets can be created in memory (due to slow handling further up or being slashdotted) the system starts a downward spiral moving memory used by applications in and out of swap. Since disk I/O comes with a severe penalty it makes sense to use the fastest storage possible (RAM, SSD and then others), use sane sysctl and application settings and rate limit excessive(!) network traffic. One level up in the web stack choosing the right web server, Nginx and Lighttpd vs Apache, has an effect on how fast requests are processed however it may have an effect on functionality so you could end up with say Nginx (or a cache like Varnish) for serving static files and Apache for dynamic content. There isn't much to say about configuration except that people often run Apache with the default MPM and all modules enabled which is bad from a security as well as a resource point of view (as every thread will require more resources than strictly necessary) and the other thing is that changing any configuration values should imply understanding what effect they have on the system as a whole. (And that goes for intricacies like the effect of using .htaccess files as well.) For example the default timeout settings may be OK for users on a board being logged in days on end but may not suit a web log visitors hit in a more "bursty" way. The same goes for the choice between the "safe choice" mod_php and FastCGI or PHP-FPM except that AFAIK Nginx doesn't run mod_php and you may really want FastCGI or PHP-FPM for some site but it may break functionality.

I don't know Boonex Dolphin or Red5 but CMS software like Joomla or WordPress may be problematic in the sense these are "one size fits all" packages. Choosing software wisely may have an effect on the application resource footprint especially if you tend to use only say ten per cent of offered functionality or less. Should you need these packages then you best first optimize them by removing or disabling components that are unnecessary, no longer maintained or resource hogs (which SEF?). For example choosing a theme that has 50 images less simply means 50 less requests to serve. And "less is more" is one of the best rules anyway, especially if you're trying to run a business. Wrt per-site-per-VM specs and performance optimization Memcache(d) and APC are standard tools and IIRC WordPress has caching plugins itself but be aware how they interact and test to find the best combination per site. SAR like Atop, Dstat and collectl, service monitoring like Monit, Zabbix and Relic (anyone know a free equivalent?), system, application level and client-side debugging tools like Firebug are your friends. And be vigilant always.

For sites that need to scale nearly instantaneously due to demand you'll want hot spares so that implies running a load balancer for easy adding servers. Add a cache or Nginx for static content, Apache for dynamic content and set up MySQL replication. Then you can XenMotion (or whatever is the equivalent of VMwares Vmotion) them across your "pool" of three XenServers according to resource usage.

1 members found this post helpful.

unSpawn. I am starting to fall in love... sorry, bad joke.
I really admire your knowledge. Any books or websites you might want to advice?

As far as I know it should be possible when installing something like MySQL to define where it is stored on your harddisk (for example on a disk in the HDD not used by anything else).
I wonder how that goes.

Same for your very interesting POV on apache vs nginx.
I have found some (outdated) tutorials how to install nginx with FCGI, but I am also afraid of the functionality of this. How would you go to make nginx only serve static content and apache the dynamic?
I am not sure how this would work.

In views on security ... All kinds actually. At this point I am running SSL on every site. The servers are secured with fail2ban, rkhunter, ossec, IPtables, custom ports & custom locations for admin panels.
(any other suggestions?)

Zabbix is free by the way. The paid variant gives you support and requests for implementations. But the main product is free and very powerful.

Boonex Dolphin was a bad choice, but I can't get away anymore. It's an OO PHP cms / social media system.
RED5 is an open source media server that provisions streaming of media (like cam, sound, ...).

The whole thing is rather a hobby of mine, not really a company (would have gone bankrupt after a month I think. I am quite bad with finances or publicity).
This way I'm hoping to learn more about CentOS, the various services, LAN and WAN network architecture, server architecture and unix server management.

No, nothing specific. Like the link below the Linux distribution you use offers basic Linux and more advanced admin documentation as does the software you use. They often have sections for security and performance tuning as well. After you've read and understood those you'll be able to tune your configuration and test the changes and after that present test results and search (try searching LQ or try say "optimize nginx wordpress" with your favorite search engine) / ask more specific questions.


That's what documentation like this is for.


If you can get away with Nginx on its own by all means test it, but where you can't or don't want to do that you can have Nginx listen on one port, have Apache listening on another port and make Nginx hand off anything it won't / can't handle itself to Apache via ProxyPass.


Looks good but I'd say it's a different topic.


...and anything else from installing lean systems (why else would CentOS offer a "minimal" installer image?) to performance tuning a system, High-Availability, provisioning and backup and disaster recovery planning (and testing). The most important thing to remember however is that fusing theoretical knowledge and practical experience together just takes time. Choosing for (any form of) virtualization means deploying hosts (using a template or Kickstart) for testing purposes can speed up the process considerably.
Read, practice, ask: but most of all just have fun with it.

with this specs and usage not pretty, just a comment friend
- 32GB ddr3 ram - I agree with the others who commented with this big of a ram build a xenserver or esx servers. so you can have a spare of vm's you can test or use on the near future.
- i5 4 core HT @ 3,4Ghz - on server i think its better to always run multicoresm, for your server i think i5-i7 processors will not suite your servers i think will it lack power and speed on the long run you always need xenon processon for your servers