How much swap area is enough/too much? Beowulf questions
Linux - SoftwareThis forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
How much swap area is enough/too much? Beowulf questions
The owner of the local computer fixit shop recently invited me to clean out his storage shed, so now I'm the proud owner of at least two dozen new-to-me Pentium 233 and 200 MHz computers. Most of the boxes are capable of running FC3, the rest RH9.
They wouldn't be worth much more than the cost of shipping, so I thought why not build a Linux Beowulf cluster supercomputer? OK, I realize how goofy it is to build a computer without actually needing it, but so what! I figure the parts needed for upgrades have depreciated about as far as they are going to--they sell Pentium CPUs by the pound to smelt the gold out of them these days. Used 10/100 NICs are less than $2 each in quantity. If I'm going to learn Linux, I might as well have something potentially useful when I'm done.
If there were such a thing as a general purpose Beowulf, that'd be what I'm aiming for. I've found a pretty good HOW-TO to follow. Now to the point of this post:
1. How much swap area is enough and how much is too much? I'm putting 128 MB RAM, 4 GB and 2 GB hard drives in each slave node. I'm using 100 MB out of the 2 GB HD for /boot. What if I devote the rest of the drive to swap? The other 4 GB HD would have one partition for the root directory. My idea is the swap area gets its own drive controller. After it's booted, swap will have exclusive use, right? I know it won't be fast, but it'll be as fast as it gets without more RAM. If I use less than the remainder of the drive, it would either be wasted or I'd have to glom part onto the 4 gig via LVM. Useful? Overmuch? Or is my mental picture of how all this works wrong?
2. I've got a variety of other more and less obsolete computers laying around here. My HOW-TO says to pick the best machine for the head node. I figure the minimum would be a PII 400 MHz, but I've got a mid-range PIII and perhaps a 1.7 GHz P4 I could use. I want to choose something fast enough to feed the slave nodes at full capacity (which probably depends greatly on how the parallel program was written), but the network is only 100 Mb/s, and it can only talk to one slave at a time, so surely it needn't be smoking lightning. Suggestions? Musings?
3. I've got a spare hardware router from when I replaced it with the firewall version. Is there any reason not to let it handle the IP addressing? Why put the network overhead on the head node? I don't have any reason to connect the cluster to the internet for now.
4. Is there some way to mirror the FC3 updates? Once I've updated one slave node, there ought to be an easy way to let all the other slave nodes get the RPMs from the one that's been updated rather than hog web bandwidth updating each node individually. Not to mention LAN vs. web download speeds....
Please, no flamers. I know I'm wasting money upgrading these dustbunny cages, but they're what I've got to work with. I admit I'm curious to know if anyone would rent the cluster for enough to pay for the electricity once I've finished it. But I'm having fun and learning a few things. If those reasons aren't enough, then surely I'll have the fastest useless computer in the world.
OK, I realize how goofy it is to build a computer without actually needing it, but so what!
Goofy? Says who? You're talking to geeks here. I suspect that many of us have some 486s, Pentiums, P2s, et.al. running for "experimental purposes". How else could we afford to have a home LAN to learn LDAP, Samba, Kerberos, clusters, etc.?
Quote:
Originally Posted by tnandy
1. How much swap area is enough and how much is too much?
Here is the REAL story about swap space. It is not possible to reliably estimate the amount of swap space that a computer will need based on RAM size or any other system feature. The ONLY way to learn what amount swap space is required on a given hardware/software system is to run the system and see what it needs under maximum load. You don't have to worry about assigning too much swap space. The worst consequence is that you would waste disk space.
Quote:
Originally Posted by tnandy
2. I've got a variety of other more and less obsolete computers laying around here. My HOW-TO says to pick the best machine for the head node. I figure the minimum would be a PII 400 MHz, but I've got a mid-range PIII and perhaps a 1.7 GHz P4 I could use. I want to choose something fast enough to feed the slave nodes at full capacity (which probably depends greatly on how the parallel program was written), but the network is only 100 Mb/s, and it can only talk to one slave at a time, so surely it needn't be smoking lightning. Suggestions? Musings?
Once again nothing beats empirical data. In other words try different things and see what happens. System administration is not theoretical. It is very much based on testing and benchmarking.
Quote:
Originally Posted by tnandy
3. I've got a spare hardware router from when I replaced it with the firewall version. Is there any reason not to let it handle the IP addressing? Why put the network overhead on the head node? I don't have any reason to connect the cluster to the internet for now.
I haven't built a Beowulf cluster so I don't know the details. Having said that it seems to me that any load that you can take off of the cluster nodes would be good. If you can use the router to assign IP addresses then great. You can (probably) use your /etc/hosts file for name/address resolution. The dedicated router will do its router stuff (IP/MAC resolution) for your cluster. It sounds like a great idea.
Quote:
Originally Posted by tnandy
Please, no flamers. I know I'm wasting money upgrading these dustbunny cages, but they're what I've got to work with.
Wouldn't dream of flaming. If you are learning something or even if it just entertains you then you are not wasting anything. You are making use of something that would otherwise end up in the land fill or computer recycler. These older machines probably use less electricity than new machines. It seems very prudent to me.
Quote:
Originally Posted by tnandy
I admit I'm curious to know if anyone would rent the cluster for enough to pay for the electricity once I've finished it.
Now I think you're dreaming, but that's part of the fun.
Last edited by stress_junkie; 01-31-2006 at 06:56 AM.
Actually from an environmental point of view, what you are doing isn't necessarily bad. There are a lot of very bad chemical and heavy metals in those computers, so you are doing a good thing by keeping them out of the landfill. When you decide to recycle them, use a responsible recycler who won't export them abroad to China or India where they turn into toxic waste and harm people's health. See my paper for info: http://www.ciber-runa.net/guide/Bett...vistGuide.html
If you can create a computer that stops you from needing to buy a new computer, then you are probably helping the environment. 80% of the total energy cost of computers comes in the original manufacturing, so you are saving the environment if you don't buy a new computer. Keeping old ones running is good, as long as you are really using them. The energy you use is less than the energy of buying one new fast computer.
Now if you really want to help the environment, the best thing is to install all those computers into a school or NGO in a thin-client network.
Still interesting to answer this question as memory , as of this moment is cheap, and running without swap is better for SSD....
In my situation I switched the linux swap completely off.
Hardware: X230 tablet, 16GB memory, SSD 1TB
Check if it is wise to disable swap completely off: $ watch free -h. The results are:
total used free shared buff/cache available
Mem: 15G 1,4G 10G 199M 3,8G 13G
Swap: 0B 0B 0B
In above free -h data I only have a few programs running. My "free" is 10GB, that means, as far as i know, 10GB is never used. Available is 13GB is my total memory that can be used. As I run new programs the "free" will go down.
If I have running: buttercup, chrome, libreoffice drawer, terminal screen, libreoffice writer and FL studio 20 with a demo song loaded, my "free" is still about 3.8GB and my free -h "available" is still 9GB. "used": 5GB
This means with 8GB it is almost possible to run without swap without problems. I have 16GB because it's cheap and some programs have memory leaks. I am pretty sure I will have no problem....
Edit:
@Jefro, post #5: I am trying to live without swap. No guts no glory!
I did find a small problem. I was working on libreoffice draw and parked the program for a moment away. Been surfing the net for info, my free -h showed my cache was getting bigger and bigger. Eventually my libreoffice draw had closed.
Been changing some memory settings, found on diverse linux websites:
sudo gedit /etc/sysctl.conf
Add the following lines:
sudo sysctl -w vm.swappiness=5;
sudo sysctl -w vm.min_free_kbytes=1000000
sudo sysctl -w vm.vfs_cache_pressure=300
Now, $ watch free -h shows my free memory is more stable and decreases way less fast. Also buff/cache (free -h) is climbing very little.
Seems to work very well the memory tuning , so far.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.