Hi everyone. I am setting up a Linux file server for a non-profit organization where I do volunteer work, and I have an interesting (well, hopefully it's interesting :-) question to share with the Linux gurus that peruse this board: which is the better option for a file server, RAMdisk or Disk Cache?

The file server's requirements are fairly simple: it will store a single, large file (a computer image) which will be sent over network connections to four separate distribution machines. These distribution machines (which are actually gutted motherboards) have been handily set up to allow easy connection of hard drives. Once operational, this system should allow for a very quick deployment of our standardized build to the hundreds of computers that will use it.

The question that has come up is whether the file server should use a RAMdisk or disk cache when serving up the image file. Several elements of the complete solution are already decided, such as installing multiple NICs on the server (so that we don't saturate a single connection), a Samba share set up with read-only access, etc. We know we need to store the image in some solid state storage, or else deal with a head flying back and forth so fast that it is creating weather patterns within a hard drive. :-) Multiple hard drives could be used to share the load, but extra maintenance would then be required when the image is updated, which happens fairly often (every couple of weeks or so)

At the outset, it seemed that disk cache was the better way to go, but after some reading I am now considering RAMdisk to be the better option. Here's a quick break down of the factors on which I am focusing:

Disk Cache

Pros
- already in place in the Linux kernel, so it requires no special configuration, reduces TCO
- no applications will be started/stopped on server, so the disk cache can be set to prioritize file caching
- dynamic process

Cons
- has a limit of 880 MB of memory for caching built in to 2.4 and 2.6 kernels...increasing this limit requires a recompile of the kernel
- as file image is already near 880 MB, it has been agreed that we would be forced to run with a custom kernel, meaning extra maintenance when upgrading kernel (i.e. for security releases)

RAMdisk

Pros
- greater control of process...know for certain that file is being served straight from memory...no possibility of data being flushed or pushed out for something else (as with disk cache)
- able to specify exact amount of RAM needed for specific job, and let the OS manage the rest efficiently

Cons
- extra setup required in writing initial scripts, creating fstab entry, etc...will need to be documented

In my research through online resources I have found a number of useful tips, including Samba and network tweaks for this type of situation which should prove useful once I am sitting in front of the console doing the final work. However, for right now, anyone who has experience with this type of setup, please share your thoughts on making something like this work, including initial setup and long term support. Also, if there is a specific Linux distro that you might recommend for this, definitely let me know!

About ramdisk: you have only to write fstab entry. Even more, it is - and has to be for glibc - on by default in /dev/shm with configuration
Maybe you need to change "defaults" to "defaults,size=1g".
It can be swapped out, though - if your box will suffer lack of physical memory.

Nope, that's tmpfs not ramdisk. Different animal.

If you have the spare ram, I'd be inclined to use the ramdisk. The (system) changes needed to accomplish this really are pretty trivial. Keep your changes to the init scripts limited to rc.local (do the mount here too), then you are less likely to be affected by system updates. Add ramdisk_size to your kernel parms in the boot loader of choice.
Done.

I would be surprised if most distros don't have the high mem option turned on in SMP 2.6 kernels these days. Especially those offered as "server" configurations.

Hadn't considered that the option for caching more than 880 MB might already be compiled into the kernel...but it does make sense for a server distro. And the computer in question for the install *is* multi-processor...hmm, I will do a bit more looking into the documentation of server distros. I had been considering using Ubuntu Server, since I used that on another server box in the place in setting up a LAMP install for a web server/inventory control system.

Just grep the config for HIGHMEM (Ubuntu desktop ship it in /boot, dunno about server)

A quick update on the RAMdisk/cache choice. After looking through the Ubuntu server config files, I did find the HIGHMEM option set to on. However, from my reading this setting only applies to whether a machine can access memory beyond 4GB. I couldn't find any setting directly affecting the aforementioned 880MB limit for cache, although there are online resources that reference this behaviour and give brief (and unsatisfying) instructions for recompiling the kernel to change it.

Anyway, after configuring the RAMdisk solution, I ran into some problems with a limit of 512MB in its size, past which the RAM drive would not mount (dmesg had some "magic..." message text, which pointed me to many others having similar issues). Online resources didn't offer a quick fix, but I was eventually able to solve this on my own by adding "kernel_blocksize=1024" to my kernel parameters (along with "kernel_size=1048576"). Then a "mke2fs -b 1024..." and the mount succeeded for a 1GB RAM drive! Yay!

However, my joy was short lived....

I started some testing by throwing several large files (around 200MB) into the RAMdisk, and performance (via md5sum calculations) was great. However, as the RAMdisk filled, I started to notice some system slowness. Then, when I tried an md5sum on a new file I received an "Out Of Memory" error and the system would no longer respond. It *appeared* to be a symptom of the files I was writing to RAMdisk also being written to the cache by the kernel, and memory not being freed/utilized properly by the cache (since so much was used by the RAM disk, which the cache may not have realized?). Googling found a few others with similar issues, but again, no solution offered.

So, since cache seemed to be wanting to store the files itself, I figured what the heck, let's give it a try. I did another reboot with the ramdisk_size commented out of the kernel parameters, and I started doing md5sum's on the files and watching "free -k" to see the cache utlization. It worked just as advertised. I was able to fill almost all of the RAM on the server as cache (up to almost 1.5GB), and the md5sum's on the files went lightning fast after the first read of each. So it appears that the 880MB limit either doesn't exist, is old information, or has been changed in newer/server kernels/distros.

In the end, the conclusion I have reached is that it is better to go with cache for the particular solution of serving up a single large file. It appears that the operating system knows better than I do (imagine that :-), and I will just have to learn to trust it, since system tweaking is problematic due to various (nefarious) issues. Yet another area where Linux and Windows are slowly and inexorably converging. Ha! ;-)

ITYM "ramdisk_", rather than "kernel_".Not a cache issue per se I believe. There used to be an issue with RAMDISK pages not being accounted properly, and so "thrashing" could start, leading to OOM getting invoked. I thought that had been fixed. I might try some tests on a spare server.
The VM is horrendiously complex - I reckon the devs do a pretty good job. Excellent actually.
No harm in playing though, can be very educational.

I would THINK that on a system with enough memory, the disk cache would be fine. I'm not aware of an 880 MB limit. Where did you find one? I've seen several articles referring to gigabytes of buffer cache.

It was one of the first documents I came upon in my search for resources:

http://gentoo-wiki.com/FAQ_Linux_Memory_Management

Now that I do a quick re-read, I have to admit that nowhere in the FAQ does it specifically refer to cache when mentioning the 880 MB limit for "accessing memory". It appears that I must have added that detail in myself...sorry for any confusion.

Looking at that, I would think any system with CONFIG_HIGHMEM should be able to use (much) >880 MB disk cache.