Linux - GeneralThis Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
My plan is to use RAM instead of a SSD to improve the performance and responsiveness of my video recorder/server.
Its main job is to record TV shows using three DVB-C cards and the VDR software and also to serve the recorded files to the client PC's.
But lately, I started to use the PC more and more for other activities like surfing the web (quite cool on the large TV screen), running my intranet web server and so on.
The system is based on Ubuntu Server 10.04 (LTS) with a 4 GB root filesystem and 8 GB of RAM. This PC runs 24/7. That's why I want to avoid a second harddisk (power consumption). But that's also why it is no problem if the soulution makes the boot process a bit slower.
Because the largest part of harddisk I/O activities occur on the video partition, the harddisk cache seems quite useless for files stored on the root filesystem.
Especially when recording multiple TV channels at once, the desktop performance of the PC is awful. This is what I want to change and why I upgraded the RAM to 8 GB. But I am very unsure what is the best way to speed it up.
These are my ideas so far, not knowing if they are practicabele at all:
Is there a way to "pin" a specific partition (root) into the harddisk cache ?
Or is there something like priorities, giving cached blocks from the root partition a much larger lifetime in the cache than blocks from other partitions ?
Or could I use a ramdisk based solution, a bit similar to how live CD's work ?
But with the difference that it needs to write through to the harddisk in case of config changes, package upgrades and so on ...
If the video partition is only for temporary storage then mounting a ramdisk on it would probably be best, it would also reduce wear on your hard drive.
None of the partitions is really for temporary storage.
The whole disk is 2 TB, the root partition is 4GB, swap another 4 GB, leaving nearly 2 TB for the video partition. Way too large for a ramdisk :-)
What I want to achieve is a system that is as responsive as if the root filesystem was on a SSD, without buying a SSD. And the responsiveness should be independent of the I/O load on the video partition.
So it's the root partition where I want to overlay a ramdisk, or pin it into the cache, or ...
But I do neither know what is the best method nor do I know how to overlay a write-through ramdisk, e.g.
Last edited by kubuntu-man; 12-04-2011 at 04:35 PM.
First thing to do is describe the problem.
To me it seems your problem is that you have 3 dvb cards writing to one disk.
How much data are you writing? If I described your problem correctly that I'd look at
adding 1, 2 or 3 hdd to the setup. You might also create a tmpfs and use that for temp dvb storage.
It all depends on your write rates.
the harddisk cache seems quite useless for files stored on the root filesystem.
You've already discovered that's not true.
There are constant updates to the kernel to improve (perceived) desktop responsiveness. Not rolled back to Lucid though.
1) no
2) no
3) yes - this is reasonably well known. The gentoo folks used to be quite into this. Haven't passed by there in a while though, so I'm not aware of the current status.
Quote:
But with the difference that it needs to write through to the harddisk in case of config changes, package upgrades and so on ...
Defeats the purpose to some degree. There are stackable filesystems, but they work the other way around.
The way it it generally handled is to write the f/s out to disk on shutdown. Maybe you could rsync it every so often - just treat it as another backup target.
Not exactly.
As long as I leave the PC alone and it does only the video recording, there is no problem. The harddisk can easily take the I/O load of the recording, even with all three DVB cards in use.
The problem comes when I do destop activities on the PC while recording. Starting programs, searching messages in my email folders, etc.
It still works, I have never lost any recordings, but the responsiveness degrades (compared to no DVB card is active) so it is just no fun working this way.
I do explicitly not want to add more harddisks because the PC runs 24/7 and so the power consumption of any additional piece of hardware will sum up.
Instead, I am looking for a way to use the RAM like it had a second extremely fast disk.
@syg00
OK, so then it will be a ramdisk.
I searched the web a bit, but I found no useful info. Or I did not understand it ...
Can you give me some useful keywords to include in my search ?
+1 to the suggestions above to use a separate hard drive for your video data. I work in the music biz and it's basic Multimedia 101 that the operating system and the data are kept separate to avoid exactly this problem. Hard drives are cheap and reliable. Sorry that's not the answer you want to hear.
You can use "preload" to put some of your most-frequently used applications into RAM, that might help a little, but it won't help in the case of surfing the web or loading emails from your /home.
First thing to do is find out what the problem is. Right now it seems you are trying to fix something that you see as broken but not sure what exactly is broken. Find your bottleneck and remove it. Would bet it's the disk.
Free things you can try:
Change the preempt model in your kernel
Change whatever is hitting the disk to instead use tmpfs
Read linux/Documentation/ as needed
Couldn't find the gentoo stuff - but that site had to be re-built a while back.
Have a look at this - will give you an idea of what is required. Read the caveats - I haven't checked the script.
Sure it's the disk. But I'm not looking for the most precisely matching solution. I'm just looking for a solution that matches my environmental requirements (no additional disk) and that needs no master degree in Linux administration.
And simulating a second disk by using RAM seems to be an easy solution. If I'm able to set it up, it will perform better than any SSD. It's "sledgehammer tactics", but that's fine with me.
90% of the I/O on the root FS is read, while the I/O on the video FS is nearly 50% read and 50% write. And 90-95% of the overall I/O is on the video FS.
But:
The video FS is nearly 2 TB. Too large for a cache or ar ramdisk.
There is a large time delay between recording a TV show and watching it (at least a few hours, sometimes even several months). These files are not short-living temporary files.
So, any action taken to optimize the video FS is completely futile !
I can only optimize the root FS !
BTW: Why do people always assume that someone who plans to do things in a bit different way has not thought about it ? I just did not want to go into details so much in my first post because these details did not matter for my question. The only would have made it harder to read and to distinguish between background info and the question.
I'll have a look at the article pointed to by syg00, then I'll decide if I overlay the whole root filesystem with the ramdisk or only parts of it. Or maybe even cancel the ramdisk idea and look for something else ...
@syg00
Thanks for the link, I will take a look at it.
First, remember that anything in RAM, including RAMdisks, will disappear if there is even a blip in the power. It's very transient, and won't survive even a half second of power outage. If it were me, and the problem only shows up when surfing, I would buy a relatively cheap netbook or even an Android tablet and use that for surfing, and let the video recorder churn away by itself. That's probably the cheapest and most effective option.
And simulating a second disk by using RAM seems to be an easy solution.
Creating a RAM disk is simple. Using it as system disk is not.
My suggestion would be to take your system partition and your /home off the disk with the video files. If you are concerned about power usage use a pendrive or SSD for that.
@sgosnell
That's why I want to use some kind of stacked write-through ramdisk and not just a ramdisk.
As writes on the root filesystem occur rarely, this will not impact the performance very much.
On the other hand, it may be even better to have just a ramdisk mounted as root and regularly sync the changes to the root partition on the harddisk (e.g. after system updates, receiving mail, ...):
In case of a power failure, the root filesystem on the harddisk can not be damaged because it is not mounted (except during sync). Then I would lose nothing but the latest syslog ...
@TobiSGD
If it turns out that I can't set up a ramdisk like I wish, a pendrive may be a plan B...
@sgosnell
As writes on the root filesystem occur rarely, this will not impact the performance very much.
If writes to the root filesystem occur rarely and do not impact performance very much, then I doubt the root filesystem is your performance bottleneck. See timetraveler's post #8: you need to run some diagnostics to understand the problem before you can look for solutions.
You already know my hypothesis what I think will be the most effective solution.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.