Ideas wanted: RAM instead of SSD
 

Hi there,

My plan is to use RAM instead of a SSD to improve the performance and responsiveness of my video recorder/server.
Its main job is to record TV shows using three DVB-C cards and the VDR software and also to serve the recorded files to the client PC's.
But lately, I started to use the PC more and more for other activities like surfing the web (quite cool on the large TV screen), running my intranet web server and so on.
The system is based on Ubuntu Server 10.04 (LTS) with a 4 GB root filesystem and 8 GB of RAM. This PC runs 24/7. That's why I want to avoid a second harddisk (power consumption). But that's also why it is no problem if the soulution makes the boot process a bit slower.

Because the largest part of harddisk I/O activities occur on the video partition, the harddisk cache seems quite useless for files stored on the root filesystem.

Especially when recording multiple TV channels at once, the desktop performance of the PC is awful. This is what I want to change and why I upgraded the RAM to 8 GB. But I am very unsure what is the best way to speed it up.

These are my ideas so far, not knowing if they are practicabele at all:

1. Is there a way to "pin" a specific partition (root) into the harddisk cache ?
2. Or is there something like priorities, giving cached blocks from the root partition a much larger lifetime in the cache than blocks from other partitions ?
3. Or could I use a ramdisk based solution, a bit similar to how live CD's work ?
   But with the difference that it needs to write through to the harddisk in case of config changes, package upgrades and so on ...

Or does someone have another idea ?

If the video partition is only for temporary storage then mounting a ramdisk on it would probably be best, it would also reduce wear on your hard drive.

Ooops, misunderstanding ...

None of the partitions is really for temporary storage.
The whole disk is 2 TB, the root partition is 4GB, swap another 4 GB, leaving nearly 2 TB for the video partition. Way too large for a ramdisk :-)

What I want to achieve is a system that is as responsive as if the root filesystem was on a SSD, without buying a SSD. And the responsiveness should be independent of the I/O load on the video partition.

So it's the root partition where I want to overlay a ramdisk, or pin it into the cache, or ...

But I do neither know what is the best method nor do I know how to overlay a write-through ramdisk, e.g.

First thing to do is describe the problem.
To me it seems your problem is that you have 3 dvb cards writing to one disk.

How much data are you writing? If I described your problem correctly that I'd look at
adding 1, 2 or 3 hdd to the setup. You might also create a tmpfs and use that for temp dvb storage.
It all depends on your write rates.

You've already discovered that's not true.
There are constant updates to the kernel to improve (perceived) desktop responsiveness. Not rolled back to Lucid though.

1) no
2) no
3) yes - this is reasonably well known. The gentoo folks used to be quite into this. Haven't passed by there in a while though, so I'm not aware of the current status.Defeats the purpose to some degree. There are stackable filesystems, but they work the other way around.
The way it it generally handled is to write the f/s out to disk on shutdown. Maybe you could rsync it every so often - just treat it as another backup target.

@timetraveler

Not exactly.
As long as I leave the PC alone and it does only the video recording, there is no problem. The harddisk can easily take the I/O load of the recording, even with all three DVB cards in use.

The problem comes when I do destop activities on the PC while recording. Starting programs, searching messages in my email folders, etc.
It still works, I have never lost any recordings, but the responsiveness degrades (compared to no DVB card is active) so it is just no fun working this way.

I do explicitly not want to add more harddisks because the PC runs 24/7 and so the power consumption of any additional piece of hardware will sum up.

Instead, I am looking for a way to use the RAM like it had a second extremely fast disk.

@syg00

OK, so then it will be a ramdisk.

I searched the web a bit, but I found no useful info. Or I did not understand it ...
Can you give me some useful keywords to include in my search ?

+1 to the suggestions above to use a separate hard drive for your video data. I work in the music biz and it's basic Multimedia 101 that the operating system and the data are kept separate to avoid exactly this problem. Hard drives are cheap and reliable. Sorry that's not the answer you want to hear. ;)

You can use "preload" to put some of your most-frequently used applications into RAM, that might help a little, but it won't help in the case of surfing the web or loading emails from your /home.

First thing to do is find out what the problem is. Right now it seems you are trying to fix something that you see as broken but not sure what exactly is broken. Find your bottleneck and remove it. Would bet it's the disk.

Free things you can try:

Change the preempt model in your kernel
Change whatever is hitting the disk to instead use tmpfs
Read linux/Documentation/ as needed

Couldn't find the gentoo stuff - but that site had to be re-built a while back.
Have a look at this - will give you an idea of what is required. Read the caveats - I haven't checked the script.

I'd start with a live usb just to see what command top would say while recording 3 shows if you want to try to take out the OS part.

My guess is you are trying to push a lot of data to the hard drive and that may be an issue of the filesystem choice for that type of data.

Might look at one of the hybrid type of hard drives too. They may get you both speed and amount.


Guess you could spring for a revodrive.

@timetraveler

Sure it's the disk. But I'm not looking for the most precisely matching solution. I'm just looking for a solution that matches my environmental requirements (no additional disk) and that needs no master degree in Linux administration.

And simulating a second disk by using RAM seems to be an easy solution. If I'm able to set it up, it will perform better than any SSD. It's "sledgehammer tactics", but that's fine with me.

90% of the I/O on the root FS is read, while the I/O on the video FS is nearly 50% read and 50% write. And 90-95% of the overall I/O is on the video FS.

But:

• The video FS is nearly 2 TB. Too large for a cache or ar ramdisk.
• There is a large time delay between recording a TV show and watching it (at least a few hours, sometimes even several months). These files are not short-living temporary files.

So, any action taken to optimize the video FS is completely futile !
I can only optimize the root FS !

BTW: Why do people always assume that someone who plans to do things in a bit different way has not thought about it ? I just did not want to go into details so much in my first post because these details did not matter for my question. The only would have made it harder to read and to distinguish between background info and the question.

I'll have a look at the article pointed to by syg00, then I'll decide if I overlay the whole root filesystem with the ramdisk or only parts of it. Or maybe even cancel the ramdisk idea and look for something else ...

@syg00
Thanks for the link, I will take a look at it.

First, remember that anything in RAM, including RAMdisks, will disappear if there is even a blip in the power. It's very transient, and won't survive even a half second of power outage. If it were me, and the problem only shows up when surfing, I would buy a relatively cheap netbook or even an Android tablet and use that for surfing, and let the video recorder churn away by itself. That's probably the cheapest and most effective option.

Creating a RAM disk is simple. Using it as system disk is not.
My suggestion would be to take your system partition and your /home off the disk with the video files. If you are concerned about power usage use a pendrive or SSD for that.

@sgosnell
That's why I want to use some kind of stacked write-through ramdisk and not just a ramdisk.
As writes on the root filesystem occur rarely, this will not impact the performance very much.

On the other hand, it may be even better to have just a ramdisk mounted as root and regularly sync the changes to the root partition on the harddisk (e.g. after system updates, receiving mail, ...):
In case of a power failure, the root filesystem on the harddisk can not be damaged because it is not mounted (except during sync). Then I would lose nothing but the latest syslog ...

@TobiSGD
If it turns out that I can't set up a ramdisk like I wish, a pendrive may be a plan B...

If writes to the root filesystem occur rarely and do not impact performance very much, then I doubt the root filesystem is your performance bottleneck. See timetraveler's post #8: you need to run some diagnostics to understand the problem before you can look for solutions.

You already know my hypothesis what I think will be the most effective solution. :)

well, you could use a kind of "persistence" that is maybe a hybrid between a HDD install and a LiveCD/USB install.

I have Slackware installed into a smallish partition, and a full Slackware install is too big, so I made a kind of hybrid system wherein the whole /usr is compressed into a squashfs filesystem which uses xz compression and requires also a aufs3 patched kernel.

the system is completely persistent as even with a poweroutage there is a read/write folder where any changes made to /usr are gathered. on boot up it decompresses the read-only usr.sqfs and then reads from the read/write folder and syncs the two, the read-write taking precedence.
yes, it needs to be re-compressed whenever you want any changes permanent but it works great.
usr uncompressed is 4.1GB and as a compressed squashfs is 1.5GB.

heres my /etc/fstab

there are other tricks you can do with aufs3 and squashfs like maybe making just a firefox module that is read-only and lives in ram, never writing to HDD or whatever.
I also make a super small Slackware -current 200mb livecd/usb that is persistent and whole thing does copy2ram ala Slax.
dont know if any of that applies but if it does or you want to try it I can help:)

see here also, though mine is kernel 3.1.4 and aufs3

https://wiki.archlinux.org/index.php...ng_Performance

http://linux.startcom.org/HOWTO/Squa...gandusing.html

http://elinux.org/Squash_FS_Howto

@snowpine
what I ment by "do not impact performance very much" is that the syncing after system updates, config changes etc. does surely impact performance remarkably for that specific moment. But it occurs so rarely that this is acceptable. And I always can postpone system upgrades until the PC is not currently recording TV shows. I should have written "overall performance" ...

Writes on the root filesystm are rare, but as soon as I use the PC interactively, reads on root aren't rare any more. Everytime I start a program or open a file, I have to wait.

I am pretty sure it's because

• the much higher I/O amount on the video partition has kicked all blocks of the root partition out of the disk cache (what absolutely makes sense from the overall system point of view, but degrades the user experience from the interactive user's point of view)
• the disk's heads are (most probably) somewhere on the video partition, causing additional seeks for reads on root

Is that enough analysis of the bottleneck ?

These two things are what I want to workaround.
Either by influencing the disk cache (I learned in this thread that's impossible)
or by decoupling the root and video partitions.

As my hard requirement is not to add another harddisk, the ramdisk seems worth a try for simulating a very fast second disk (I know I wrote that before).

Gernal note:
The PC (and especially the harddisk) is not operating on razor's edge. It just "feels" uncomfortably slow to the interactive user.

This project is a partially about improving the user experience (which has a positive influence on the the WAF, so it's absolutely not unimportant) and partially about finding out if it is doable.

@linus72
Hey, cool, this is something I will also have a look at.

Sadly, my boss asked me today to work overtime for the rest of the year. So my work on this project will be much slower than I have planned as I will have very few free time from now on ... :(


[Edit]
WAF = Woman Acceptance Factor
[/Edit]

If it's simply a matter of a "snappy" user interface then have you experimented with: disabling desktop effects, using a "lightweight" DE like LXDE, optimizing graphics drivers/swappiness/startup services, etc?

The desktop is as minimalistic as it can be: OpenBox.

Well, if you want to run your system from RAM this is the approach that works on Slackware. Since Ubuntu uses a different init-system I don't know if that will work on Ubuntu.