How to disable filesystem cache?

nawwar · 03-10-2008, 12:20 PM

Hello,
I need to do some speed tests and benchmarks on my hard disk drive. Unfortunately the files being written/read are being cached, (normally it's a good thing, but not when I need to accurately measure the performance of the harddisk). I looked in 'mount' options, 'hdparm', even the kernel configuration. Unfortunately I couldn't find anything. I feel that it should be a simple switch somewhere, but I'm just missing it.
Please, if any one knows how to disable the cache, I'd appreciate the help. Even if it takes to hack the kernel, I have no problem.
Thanks in advance,
Nawwar

PS. I hope I put the question in the correct forum!

raskin · 03-10-2008, 02:24 PM

Does passing sync,dirsync flags to mount do what you need? For testing a new disk, I'd use dd between it and some file on tmpfs, initialized from /dev/urandom (surely it would overwrite all partition tables and any data on it, so write speed should be tested like that only for empty disks).

nawwar · 03-10-2008, 02:35 PM

Thanks raskin for the reply.
Although sync option will allow me bypass the cache when writing, the cache is still used for reading. So unfortunately it is not enough. I need read commands to read from the disk.

Edit:
And no, it's not a new disk, I do need the file system for my tests!

raskin · 03-10-2008, 02:48 PM

Well, preventing read cache usage is pretty simple. You can do some raw-read tests with dd. You can also do some tests (one-time read only, unfortunately, using drop_caches):

Code:

drop_caches
-----------

Writing to this will cause the kernel to drop clean caches, dentries and
inodes from memory, causing that memory to become free.

To free pagecache:
	echo 1 > /proc/sys/vm/drop_caches
To free dentries and inodes:
	echo 2 > /proc/sys/vm/drop_caches
To free pagecache, dentries and inodes:
	echo 3 > /proc/sys/vm/drop_caches

As this is a non-destructive operation and dirty objects are not freeable, the
user should run `sync' first.

(see linux/Documentation/filesystems/proc.txt).

nawwar · 03-10-2008, 03:25 PM

Thanks again for your help. Actually this is really interesting, it's a new way that I didn't know about.
Actually I also know another technique too, it is to unmount/remount the device. However, both ways will free the cache for once. In my test I may read the same block twice in the same test! Anyway, i think I can work-around this point and make sure no reading is done twice on the same block (although this test may lose some value for certain reason!).
But still, if you, or anyone, know a way to completely disable the cache, it would save me a lot of hassle.
If there is no such a was as disabling the cache in linux. Then I think I'll have to go for limiting my test a bit.

Thank you very much again

raskin · 03-10-2008, 03:43 PM

What amount of data are we speaking about? Keep a low-CPU memor-eater running, and cache will die off on its own... See Documentation directory in kernel sources to find out how to make programs always take over cache.. I do not know, but probably finding cache lookup in VFS code and making them always fail is not hard.

nawwar · 03-10-2008, 03:58 PM

I'm speaking about 100~400MB, but it is worth mentioning that the access is actually somehow random no sequential. (In some cases I may access some blocks frequently). It is actually a complicated test, that's really the problem.
I guess memory eater process may be what I need, it should resolve the problem. I think i'll have to disable the swap though, it's IOs may affect the results of my test.
And yes, I'll take a look at the VFS code first.
Thanks again

trailman · 10-20-2008, 09:43 AM

Hello,

Thanks raskin for your trick on /proc/sys/vm/drop_caches.
I have wrote a simple shell script to bench my disk.

Code:

FILENAME=/tmp/test.tmp
BLOCKSIZE=1M
BLOCKCOUNT=100

echo '--- Write ---'
dd if=/dev/zero of=$FILENAME bs=$BLOCKSIZE count=$BLOCKCOUNT conv=fsync

echo 3 > /proc/sys/vm/drop_caches
echo 0 > /proc/sys/vm/drop_caches

echo '---Read ---'
dd if=$FILENAME of=/dev/zero bs=$BLOCKSIZE count=$BLOCKCOUNT

rm -f $FILENAME

Quakeboy02 · 10-20-2008, 11:29 PM

Have you considered using bonnie for your benchmarks? Install package bonnie++.

trailman · 10-22-2008, 04:19 AM

In fact, I have not used bonnie or another bench because my customer wants a very very small application to play with.

Fortunately, using some C source code, there's a way to do write and read without cache effects : use open() with the O_DIRECT option

This requires some alignment on the buffer passed to read() or write(). See open man page.
Additionally, the following define may be required before all #include lines : #define _GNU_SOURCE

gam2102 · 04-10-2009, 10:19 AM

Following up on the previous posts, I wrote a little script to continuously flush the cache:

Code:

#!/bin/bash
for ((i = 1; i < 1000000; i++)); do
   echo 1 > /proc/sys/vm/drop_caches
   sleep 1
done

Thanks everybody.

George

raskin · 04-10-2009, 12:35 PM

"while true; do" is a better expression of infinite loop than "for ((i = 1; i < 1000000; i++)); do"

Quakeboy02 · 04-10-2009, 02:31 PM

Quote:

Originally Posted by trailman

In fact, I have not used bonnie or another bench because my customer wants a very very small application to play with.

Smaller than 102K? That's more or less the installed size of bonnie++.

trailman · 04-19-2009, 03:44 PM

Quote:

Originally Posted by Quakeboy02

Smaller than 102K? That's more or less the installed size of bonnie++.

Yes, you're right ! ... and at the end I used bonnie++ 1.03e for my bench.
I've used the option for the O_DIRECT mode
I get nearly the same bench results than with my previous bench script.

syg00 · 04-19-2009, 04:39 PM

Quote:

Originally Posted by gam2102

Following up on the previous posts, I wrote a little script to continuously flush the cache:

And why would you want to do that ?.
The OP had a specific reason for bypassing cache - in general use, file caching is a (significant) benefit.
Running your little script exposes you to significant risk of data loss.