LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 12-04-2011, 03:22 PM   #1
kubuntu-man
LQ Newbie
 
Registered: Oct 2011
Posts: 27

Rep: Reputation: Disabled
Ideas wanted: RAM instead of SSD


Hi there,

My plan is to use RAM instead of a SSD to improve the performance and responsiveness of my video recorder/server.
Its main job is to record TV shows using three DVB-C cards and the VDR software and also to serve the recorded files to the client PC's.
But lately, I started to use the PC more and more for other activities like surfing the web (quite cool on the large TV screen), running my intranet web server and so on.
The system is based on Ubuntu Server 10.04 (LTS) with a 4 GB root filesystem and 8 GB of RAM. This PC runs 24/7. That's why I want to avoid a second harddisk (power consumption). But that's also why it is no problem if the soulution makes the boot process a bit slower.

Because the largest part of harddisk I/O activities occur on the video partition, the harddisk cache seems quite useless for files stored on the root filesystem.

Especially when recording multiple TV channels at once, the desktop performance of the PC is awful. This is what I want to change and why I upgraded the RAM to 8 GB. But I am very unsure what is the best way to speed it up.

These are my ideas so far, not knowing if they are practicabele at all:
  1. Is there a way to "pin" a specific partition (root) into the harddisk cache ?
  2. Or is there something like priorities, giving cached blocks from the root partition a much larger lifetime in the cache than blocks from other partitions ?
  3. Or could I use a ramdisk based solution, a bit similar to how live CD's work ?
    But with the difference that it needs to write through to the harddisk in case of config changes, package upgrades and so on ...
Or does someone have another idea ?
 
Old 12-04-2011, 03:54 PM   #2
kbp
Senior Member
 
Registered: Aug 2009
Posts: 3,758

Rep: Reputation: 643Reputation: 643Reputation: 643Reputation: 643Reputation: 643Reputation: 643
If the video partition is only for temporary storage then mounting a ramdisk on it would probably be best, it would also reduce wear on your hard drive.
 
Old 12-04-2011, 04:19 PM   #3
kubuntu-man
LQ Newbie
 
Registered: Oct 2011
Posts: 27

Original Poster
Rep: Reputation: Disabled
Ooops, misunderstanding ...

None of the partitions is really for temporary storage.
The whole disk is 2 TB, the root partition is 4GB, swap another 4 GB, leaving nearly 2 TB for the video partition. Way too large for a ramdisk :-)

What I want to achieve is a system that is as responsive as if the root filesystem was on a SSD, without buying a SSD. And the responsiveness should be independent of the I/O load on the video partition.

So it's the root partition where I want to overlay a ramdisk, or pin it into the cache, or ...

But I do neither know what is the best method nor do I know how to overlay a write-through ramdisk, e.g.

Last edited by kubuntu-man; 12-04-2011 at 04:35 PM.
 
Old 12-04-2011, 04:49 PM   #4
timetraveler
Member
 
Registered: Apr 2010
Posts: 243
Blog Entries: 2

Rep: Reputation: 31
First thing to do is describe the problem.
To me it seems your problem is that you have 3 dvb cards writing to one disk.

How much data are you writing? If I described your problem correctly that I'd look at
adding 1, 2 or 3 hdd to the setup. You might also create a tmpfs and use that for temp dvb storage.
It all depends on your write rates.
 
Old 12-04-2011, 05:08 PM   #5
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,354

Rep: Reputation: 1042Reputation: 1042Reputation: 1042Reputation: 1042Reputation: 1042Reputation: 1042Reputation: 1042Reputation: 1042
Quote:
Originally Posted by kubuntu-man View Post
the harddisk cache seems quite useless for files stored on the root filesystem.
You've already discovered that's not true.
There are constant updates to the kernel to improve (perceived) desktop responsiveness. Not rolled back to Lucid though.

1) no
2) no
3) yes - this is reasonably well known. The gentoo folks used to be quite into this. Haven't passed by there in a while though, so I'm not aware of the current status.
Quote:
But with the difference that it needs to write through to the harddisk in case of config changes, package upgrades and so on ...
Defeats the purpose to some degree. There are stackable filesystems, but they work the other way around.
The way it it generally handled is to write the f/s out to disk on shutdown. Maybe you could rsync it every so often - just treat it as another backup target.
 
Old 12-05-2011, 10:56 AM   #6
kubuntu-man
LQ Newbie
 
Registered: Oct 2011
Posts: 27

Original Poster
Rep: Reputation: Disabled
@timetraveler

Not exactly.
As long as I leave the PC alone and it does only the video recording, there is no problem. The harddisk can easily take the I/O load of the recording, even with all three DVB cards in use.

The problem comes when I do destop activities on the PC while recording. Starting programs, searching messages in my email folders, etc.
It still works, I have never lost any recordings, but the responsiveness degrades (compared to no DVB card is active) so it is just no fun working this way.

I do explicitly not want to add more harddisks because the PC runs 24/7 and so the power consumption of any additional piece of hardware will sum up.

Instead, I am looking for a way to use the RAM like it had a second extremely fast disk.

@syg00

OK, so then it will be a ramdisk.

I searched the web a bit, but I found no useful info. Or I did not understand it ...
Can you give me some useful keywords to include in my search ?
 
Old 12-05-2011, 11:52 AM   #7
snowpine
Senior Member
 
Registered: Feb 2009
Posts: 3,918

Rep: Reputation: 1049Reputation: 1049Reputation: 1049Reputation: 1049Reputation: 1049Reputation: 1049Reputation: 1049Reputation: 1049
+1 to the suggestions above to use a separate hard drive for your video data. I work in the music biz and it's basic Multimedia 101 that the operating system and the data are kept separate to avoid exactly this problem. Hard drives are cheap and reliable. Sorry that's not the answer you want to hear.

You can use "preload" to put some of your most-frequently used applications into RAM, that might help a little, but it won't help in the case of surfing the web or loading emails from your /home.
 
Old 12-05-2011, 03:28 PM   #8
timetraveler
Member
 
Registered: Apr 2010
Posts: 243
Blog Entries: 2

Rep: Reputation: 31
First thing to do is find out what the problem is. Right now it seems you are trying to fix something that you see as broken but not sure what exactly is broken. Find your bottleneck and remove it. Would bet it's the disk.

Free things you can try:

Change the preempt model in your kernel
Change whatever is hitting the disk to instead use tmpfs
Read linux/Documentation/ as needed
 
Old 12-05-2011, 04:49 PM   #9
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,354

Rep: Reputation: 1042Reputation: 1042Reputation: 1042Reputation: 1042Reputation: 1042Reputation: 1042Reputation: 1042Reputation: 1042
Couldn't find the gentoo stuff - but that site had to be re-built a while back.
Have a look at this - will give you an idea of what is required. Read the caveats - I haven't checked the script.
 
Old 12-05-2011, 05:37 PM   #10
jefro
Guru
 
Registered: Mar 2008
Posts: 11,742

Rep: Reputation: 1445Reputation: 1445Reputation: 1445Reputation: 1445Reputation: 1445Reputation: 1445Reputation: 1445Reputation: 1445Reputation: 1445Reputation: 1445
I'd start with a live usb just to see what command top would say while recording 3 shows if you want to try to take out the OS part.

My guess is you are trying to push a lot of data to the hard drive and that may be an issue of the filesystem choice for that type of data.

Might look at one of the hybrid type of hard drives too. They may get you both speed and amount.


Guess you could spring for a revodrive.
 
Old 12-05-2011, 06:11 PM   #11
kubuntu-man
LQ Newbie
 
Registered: Oct 2011
Posts: 27

Original Poster
Rep: Reputation: Disabled
@timetraveler

Sure it's the disk. But I'm not looking for the most precisely matching solution. I'm just looking for a solution that matches my environmental requirements (no additional disk) and that needs no master degree in Linux administration.

And simulating a second disk by using RAM seems to be an easy solution. If I'm able to set it up, it will perform better than any SSD. It's "sledgehammer tactics", but that's fine with me.

90% of the I/O on the root FS is read, while the I/O on the video FS is nearly 50% read and 50% write. And 90-95% of the overall I/O is on the video FS.

But:
  • The video FS is nearly 2 TB. Too large for a cache or ar ramdisk.
  • There is a large time delay between recording a TV show and watching it (at least a few hours, sometimes even several months). These files are not short-living temporary files.
So, any action taken to optimize the video FS is completely futile !
I can only optimize the root FS !

BTW: Why do people always assume that someone who plans to do things in a bit different way has not thought about it ? I just did not want to go into details so much in my first post because these details did not matter for my question. The only would have made it harder to read and to distinguish between background info and the question.

I'll have a look at the article pointed to by syg00, then I'll decide if I overlay the whole root filesystem with the ramdisk or only parts of it. Or maybe even cancel the ramdisk idea and look for something else ...

@syg00
Thanks for the link, I will take a look at it.
 
Old 12-05-2011, 06:43 PM   #12
sgosnell
Member
 
Registered: Jan 2008
Location: Baja Oklahoma
Distribution: Debian
Posts: 400

Rep: Reputation: 68
First, remember that anything in RAM, including RAMdisks, will disappear if there is even a blip in the power. It's very transient, and won't survive even a half second of power outage. If it were me, and the problem only shows up when surfing, I would buy a relatively cheap netbook or even an Android tablet and use that for surfing, and let the video recorder churn away by itself. That's probably the cheapest and most effective option.
 
Old 12-05-2011, 10:57 PM   #13
TobiSGD
Moderator
 
Registered: Dec 2009
Location: Hanover, Germany
Distribution: Main: Gentoo Others: What fits the task
Posts: 15,592
Blog Entries: 2

Rep: Reputation: 4047Reputation: 4047Reputation: 4047Reputation: 4047Reputation: 4047Reputation: 4047Reputation: 4047Reputation: 4047Reputation: 4047Reputation: 4047Reputation: 4047
Quote:
Originally Posted by kubuntu-man View Post
And simulating a second disk by using RAM seems to be an easy solution.
Creating a RAM disk is simple. Using it as system disk is not.
My suggestion would be to take your system partition and your /home off the disk with the video files. If you are concerned about power usage use a pendrive or SSD for that.
 
Old 12-07-2011, 01:50 PM   #14
kubuntu-man
LQ Newbie
 
Registered: Oct 2011
Posts: 27

Original Poster
Rep: Reputation: Disabled
@sgosnell
That's why I want to use some kind of stacked write-through ramdisk and not just a ramdisk.
As writes on the root filesystem occur rarely, this will not impact the performance very much.

On the other hand, it may be even better to have just a ramdisk mounted as root and regularly sync the changes to the root partition on the harddisk (e.g. after system updates, receiving mail, ...):
In case of a power failure, the root filesystem on the harddisk can not be damaged because it is not mounted (except during sync). Then I would lose nothing but the latest syslog ...

@TobiSGD
If it turns out that I can't set up a ramdisk like I wish, a pendrive may be a plan B...
 
Old 12-07-2011, 01:58 PM   #15
snowpine
Senior Member
 
Registered: Feb 2009
Posts: 3,918

Rep: Reputation: 1049Reputation: 1049Reputation: 1049Reputation: 1049Reputation: 1049Reputation: 1049Reputation: 1049Reputation: 1049
Quote:
Originally Posted by kubuntu-man View Post
@sgosnell
As writes on the root filesystem occur rarely, this will not impact the performance very much.
If writes to the root filesystem occur rarely and do not impact performance very much, then I doubt the root filesystem is your performance bottleneck. See timetraveler's post #8: you need to run some diagnostics to understand the problem before you can look for solutions.

You already know my hypothesis what I think will be the most effective solution.

Last edited by snowpine; 12-07-2011 at 02:00 PM.
 
  


Reply

Tags
cache, performance


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
SSD and 8GB RAM, What size SWAP? wulp Linux - General 9 08-25-2011 10:38 AM
socket connection. Ideas wanted vortmax Programming 1 06-16-2008 06:16 PM
Network monitoring Ideas wanted here2serve Linux - Enterprise 2 03-27-2008 06:47 PM
ideas wanted please frieza General 3 02-11-2005 09:37 PM
WANTED: Your ideas for Rescue Linux scuzzman Linux - General 2 12-27-2004 04:19 AM


All times are GMT -5. The time now is 10:36 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration