LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 11-23-2012, 08:50 PM   #1
Rookie1337
LQ Newbie
 
Registered: Nov 2012
Posts: 13

Rep: Reputation: Disabled
Question Trying to use ramdisks/tmpfs to speed things up


I'm trying to speed Linux up as much as possible and found this post from a long time again (2009) talking about using unionfs to merge some dirs like /lib /bin into dirs stored in RAM for crazy speed increases.

http://ubuntuforums.org/showpost.php...27&postcount=1

Unfortunately, unionfs is dead and buried as far as I know and I can't figure out if I can get overlayfs to reproduce this method. Does someone know if there's a way to have overlayfs work the same way?

I had another idea which just used mount -B to link /lib /bin /share dirs to /run/shm but again don't know if it would work.

Any help is appreciated and thanks in advance.

[EDITED] The solution is my last post. There's no real way apparently to do what I thought could be done so this thread is solved/over. Thanks.

Last edited by Rookie1337; 11-26-2012 at 03:41 PM.
 
Old 11-24-2012, 03:41 PM   #2
Elv13
Member
 
Registered: Apr 2006
Location: Montreal,Quebec
Distribution: Gentoo
Posts: 825

Rep: Reputation: 128Reputation: 128
Hi, welcome to LQ.

Linux will automagically cache a lot of files in memory by default so you don't have to do this. If you have a lot of ram, a very common technique, sometime used by distributions by default is read-ahead or preloading.

The principles are simple, during or quickly after booting. You cat "/path/to/files > /dev/null" all files you read often, including libraries and applications. Doing, that, you add them to the in-memory file cache. So it will basically be ready when you need them, just as your solution. You can then gather statistics using a daemon watching file descriptors to know which file to load first and so on. This do give some crazy speed too. There is already some daemon like ureadahead to do this.

An other solution, more closely related to the original one is to use LZO compressed ramdisks. That way you can store more information into less memory. I am not sure if this would be faster, I don't think so. Ramdisks are amazing, but to store something as big as /usr/lib, no so sure. I guess you could also do one huge initramfs image with most of the system already loaded and keep it as root. I guess this could work, but it would take 10 minutes to boot your computer.
 
1 members found this post helpful.
Old 11-24-2012, 05:25 PM   #3
Rookie1337
LQ Newbie
 
Registered: Nov 2012
Posts: 13

Original Poster
Rep: Reputation: Disabled
I'm on Kubuntu 12.10 and I'm sure it has either/both ureadahead and readahead but the problem is their defaults never seem to do anything as I'm still hitting the HDDs hard on the first load of a program and the load times aren't that great after that (they're not like the ramdisk setup I have in Windows please don't hurt me for mentioning that). I've tried preload and prelink in the past and they both didn't seem to really do much. I think time showed a whole 1second improvement in xbmc which went from something like 5 seconds to 4...which still doesn't match my other OS's speed(and I hate that I've made it faster then Linux).

Hmm...that's the second time someone has mentioned cat "src" > /dev/null and I'm wondering why. I thought /dev/null was the "black hole" of Linux systems in which once things went there then they were gone. But then comes the big problem...how can I hold the things that I want in the cache so the kernel doesn't dump them?

I wrote up a script that seemed to work until the first line of free dropped to 500 in the readout (the part the system claims is free not the "actual" free amount). Then my overlayfs system seemed to slow down.

Here's my attempt...tell me if it's stupid or pointless. I couldn't decide between binding and overlayfs. I've only tried overlayfs with this script so far so IDK if binding would be any better. I'm going to try out your method.

Quote:
#!/bin/bash
#Must chmod +x first
#Binding dirs to Rapiddisk
#You need to have made the Rapiddisk modules which required the kernel source

sudo rm -rf /mnt/Rapiddisk
sudo modprobe rxdsk
sudo rxadm --attach 4096
#creates 4GB ramdisk at /dev/rxd0
sudo mke2fs -F /dev/rxd0
sudo mkdir /mnt/Rapiddisk
sudo mount /dev/rxd0 /mnt/Rapiddisk

#First is to make dirs
sudo mkdir /mnt/Rapiddisk/google
sudo mkdir /mnt/Rapiddisk/usrlib
sudo mkdir /mnt/Rapiddisk/lib
sudo mkdir /mnt/Rapiddisk/bin
sudo mkdir /mnt/Rapiddisk/share

#Next is to sync dirs
sudo rsync -aEvz /usr/bin /mnt/Rapiddisk/bin
sudo rsync -aEvz /opt/google/chrome /mnt/Rapiddisk/google
sudo rsync -aEvz /usr/lib /mnt/Rapiddisk/usrlib
sudo rsync -aEvz /usr/share /mnt/Rapiddisk/share
sudo rsync -aEvz /lib /mnt/Rapiddisk/lib

#Finally binding
#sudo mount -B /opt/google/chrome /mnt/Rapiddisk/google
#sudo mount -B /usr/lib /mnt/Rapiddisk/usrlib
#sudo mount -B /usr/share /mnt/Rapiddisk/share
#sudo mount -B /usr/bin /mnt/Rapiddisk/bin
#sudo mount -B /lib /mnt/Rapiddisk/lib

#Finally Mount overlayfs
sudo mount -t overlayfs overlayfs -olowerdir=/opt/google/chrome,upperdir=/mnt/Rapiddisk/google /mnt/Rapiddisk/google
sudo mount -t overlayfs overlayfs -olowerdir=/usr/lib,upperdir=/mnt/Rapiddisk/usrlib /mnt/Rapiddisk/usrlib
sudo mount -t overlayfs overlayfs -olowerdir=/usr/bin,upperdir=/mnt/Rapiddisk/bin /mnt/Rapiddisk/bin
sudo mount -t overlayfs overlayfs -olowerdir=/usr/share,upperdir=/mnt/Rapiddisk/share /mnt/Rapiddisk/share
sudo mount -t overlayfs overlayfs -olowerdir=/lib,upperdir=/mnt/Rapiddisk/lib /mnt/Rapiddisk/lib
 
Old 11-24-2012, 05:39 PM   #4
Elv13
Member
 
Registered: Apr 2006
Location: Montreal,Quebec
Distribution: Gentoo
Posts: 825

Rep: Reputation: 128Reputation: 128
The kernel wont dump things unless it need the memory. If it need the memory, then you are in trouble with your ramdisks. It will move them to the swap, making things slower. ureadahead have a static filelist. You need to update this list of you want other binaries preloaded. You can also get an SSD.
 
1 members found this post helpful.
Old 11-24-2012, 06:13 PM   #5
Rookie1337
LQ Newbie
 
Registered: Nov 2012
Posts: 13

Original Poster
Rep: Reputation: Disabled
I am getting an SSD soon. I guess until then I have to settle with Windows being faster than Linux? I find it depressing that I've made it so Windows on a HDD is faster than Linux on the same HDD. I really wanted to find a way to fix that problem.

But anyway...say I wanted to move /usr/lib into the fs cache how could I do that since cat won't let me? Also...my readahead/ureadahead have empty .confs and I'm not sure how to use them for my intentions of pulling certain dirs into cache as they don't offer much direction in the manpages and seem focused on only being for speeding up booting.
 
Old 11-25-2012, 03:34 AM   #6
Elv13
Member
 
Registered: Apr 2006
Location: Montreal,Quebec
Distribution: Gentoo
Posts: 825

Rep: Reputation: 128Reputation: 128
Windows do preload most used DLLs the same way Linux does, but Linux list is static while Windows one is dynamic. Make the Linux list dynamic and it will be as fast. Just as with Windows, boot time will slow down over time (yes, this is why Windows boot is always slower).
 
Old 11-25-2012, 10:50 AM   #7
Rookie1337
LQ Newbie
 
Registered: Nov 2012
Posts: 13

Original Poster
Rep: Reputation: Disabled
Umm...you lost me there...how could I go about doing what you just said? Is that in readahead/ureadahead or preload?
 
Old 11-25-2012, 12:20 PM   #8
jefro
Moderator
 
Registered: Mar 2008
Posts: 15,374

Rep: Reputation: 2198Reputation: 2198Reputation: 2198Reputation: 2198Reputation: 2198Reputation: 2198Reputation: 2198Reputation: 2198Reputation: 2198Reputation: 2198Reputation: 2198
If you boot to a pre os then copy your filesystem to a ramdrive or such then boot to it, you will get the so called crazy speeds. Many of the minimal type live cd's still do such a trick. They tend to use a squashfs file to copy to ram. They also use aufs or union or whatever is current.
 
Old 11-25-2012, 01:13 PM   #9
Rookie1337
LQ Newbie
 
Registered: Nov 2012
Posts: 13

Original Poster
Rep: Reputation: Disabled
Yeah I'm familiar with that distro type as SLAX was my first foray into Linux back in 2010. And I am hoping that SLAX7 will be as easy as 6 was because I might just end up using it as the primary.

But the methods of trying to do such a thing with other distros are either extremely complicated and/or outdated. And in my first post someone showed how you could use unionfs to basically have everything you put into the unionfs read from ram. But since unionfs and aufs are dead/purged from the mainline kernel I was originally looking to see if there is a way to use overlayfs to replicate that method. As far as I have tried nothing really works. I think (if I understand Linux's fs_cache) I finally realized that the reason I was seeing speed improvements was not because the overlayfs or bind setup I had like in the script were actually doing what I wanted them too; but rather because I had invoked a copy command (rsync) and that it was filling the cache. Hence, eventually things would slow down as the cache began to change as I continued using the computer.

I had another (likely bad/impossible) idea as to how to get the things I wanted stored in ram but I figured it wouldn't work as it would involve copying the desired dirs to some temporary location (to store the info until it can be moved) and then mounting their original location into a Rapiddisk/ramfs/tmpfs and then copying their contents to the original locations that have now be mounted as a ram location. Example copy the contents of /usr/lib to some location; mount /usr/lib into a ram location; move the copied contents back to /usr/lib; enjoy the speed of instant loading programs. Sadly, I'm pretty sure this wouldn't work but maybe someone can tell me with certainty?
 
Old 11-25-2012, 01:53 PM   #10
TobiSGD
Moderator
 
Registered: Dec 2009
Location: Germany
Distribution: Whatever fits the task best
Posts: 17,130
Blog Entries: 2

Rep: Reputation: 4825Reputation: 4825Reputation: 4825Reputation: 4825Reputation: 4825Reputation: 4825Reputation: 4825Reputation: 4825Reputation: 4825Reputation: 4825Reputation: 4825
Quote:
Originally Posted by Rookie1337 View Post
But the methods of trying to do such a thing with other distros are either extremely complicated and/or outdated.
I have done that with Slackware and AUFS and actually it was pretty easy. I was using AUFS at that time, but I see no reason why that shouldn't work with overlayfs. I gave up on this approach for simple reasons:
1. Using a fast SSD gave me a good performance enhancement already.
2. I switched to using mostly very lightweight applications that are started automatically when the system boots.
3. With nowadays RAM sizes it is not likely that you really suffer from a to small cache.

The effort you have to make outweighs the benefits, at least for me.
 
Old 11-25-2012, 03:42 PM   #11
Rookie1337
LQ Newbie
 
Registered: Nov 2012
Posts: 13

Original Poster
Rep: Reputation: Disabled
Well, I thought aufs and unionfs were dead or removed from the mainline kernels and most of the methods for changing say slackware or buntu are beyond me as I'm just a simple general user and the level of detail I can handle is basically demonstrated by the (I guess useless) script I posted earlier(AKA note much). But your point #3 is something that has me confused...on first boot my cache from free -m is tiny and it only ever grows after I've started programs. What I want is that speed on the second start for the first start...so I want the cache filled and retaining what I want all the time. But so far I've found nothing that seems to be able to do that. That's what led me to seek out the ramfs/tmpfs/ramdisk/overlayfs ideas.

So basically, there isn't any way besides getting an SSD or using some premade toram distros like SLAX or Porteus to really make things faster for a general user or someone with limited time?
 
Old 11-25-2012, 03:51 PM   #12
TobiSGD
Moderator
 
Registered: Dec 2009
Location: Germany
Distribution: Whatever fits the task best
Posts: 17,130
Blog Entries: 2

Rep: Reputation: 4825Reputation: 4825Reputation: 4825Reputation: 4825Reputation: 4825Reputation: 4825Reputation: 4825Reputation: 4825Reputation: 4825Reputation: 4825Reputation: 4825
Quote:
Originally Posted by Rookie1337 View Post
Well, I thought aufs and unionfs were dead or removed from the mainline kernels
There are patches at least for AUFS, I think I will have a look at overlayfs and see if that is easier.

Quote:
What I want is that speed on the second start for the first start...so I want the cache filled and retaining what I want all the time.
Just use preload for that, it will keep track of the most used programs and load the needed libraries when the computer is idling/at boot. This comes with the downside of longer boot times, since at one time the libraries/programs have to be loaded into RAM.
Or just do it like I do, if you have plenty of RAM (I have 16GB on my main machine) just autostart your applications at boot.
 
Old 11-25-2012, 04:18 PM   #13
Rookie1337
LQ Newbie
 
Registered: Nov 2012
Posts: 13

Original Poster
Rep: Reputation: Disabled
Alright. I'll retry preload but how can I tell if it is actually doing anything? The last time I had it installed it didn't change any of the cache or free values so was it not running?

And how do you have it set to start up your programs on boot? Just have them running when you shutdown? Or do you have something else going on? It doesn't help me with a program like XBMC unless there's another method besides just leaving programs open...man there's so much I really don't understand about how Linux works.
 
Old 11-25-2012, 05:58 PM   #14
Elv13
Member
 
Registered: Apr 2006
Location: Montreal,Quebec
Distribution: Gentoo
Posts: 825

Rep: Reputation: 128Reputation: 128
In theory, if you have enough RAM, the second start should always be preloaded unless you flush the cache. Firefox and LibreOffice are just slow to start, there is nothing you can do about it. Some apps are faster on Windows because they enable PGO. Most Linux distribution don't do any PGO at all, so it is not fair comparison. If you want to enable profile guided optimization, startup time optimization and reducing startup time dependencies mesh, then the you need Gentoo. Even then, you need Gentoo and a lot of time to perform tens of 1-2% optimizations.
 
1 members found this post helpful.
Old 11-25-2012, 10:36 PM   #15
Rookie1337
LQ Newbie
 
Registered: Nov 2012
Posts: 13

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Elv13 View Post
In theory, if you have enough RAM, the second start should always be preloaded unless you flush the cache. Firefox and LibreOffice are just slow to start, there is nothing you can do about it. Some apps are faster on Windows because they enable PGO. Most Linux distribution don't do any PGO at all, so it is not fair comparison. If you want to enable profile guided optimization, startup time optimization and reducing startup time dependencies mesh, then the you need Gentoo. Even then, you need Gentoo and a lot of time to perform tens of 1-2% optimizations.
Hmm...I never knew/heard about PGO but it didn't seem to help windows that much. My problem is that in Windows I have the option to install programs to a savable ramdisk and as such they all start as fast as ram speeds allow. I find it ironic that the "backwards" compartmentalized way of windows actually turned into something useful.

Alright, so in your opinion are preload and even prelink worth while? Do they even work?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
little tricks to speed things up... AndrewJS Linux - Newbie 11 06-20-2011 12:07 AM
/etc/fstab "tmpfs /tmp tmpfs defaults 0 0" ramfs?! HOW? xcislav Linux - General 2 06-04-2011 04:03 AM
LXer: Speed up Firefox by mounting the profile in tmpfs (RAM) [Linux only] LXer Syndicated Linux News 0 05-04-2009 10:11 AM
Speed things up? brunog Fedora 2 11-25-2005 12:26 PM
I want to speed up Red Hat things up but how? jimdaworm Red Hat 10 09-08-2003 02:52 PM


All times are GMT -5. The time now is 02:49 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration