LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - General (http://www.linuxquestions.org/questions/linux-general-1/)
-   -   Need help interpreting my filesystem benchmark data (http://www.linuxquestions.org/questions/linux-general-1/need-help-interpreting-my-filesystem-benchmark-data-900888/)

H_TeXMeX_H 09-03-2011 10:13 AM

Need help interpreting my filesystem benchmark data
 
I decided to do my own filesystem benchmarking, because I didn't trust Phoronix.

So let me explain what I did, I booted the Slackware64 13.37 install disk, mounted a USB stick, formatted one of my partitions as various filesystems, and I mount it and ran a benchmarking script.

First the commands I ran:

Code:

mkfs.ext4 /dev/sda3
mount -o noatime /dev/sda3 /mnt/disk
df > /mnt/usb/df.ext4
cd /mnt/disk
mkdir test
cd test
echo cfq > /sys/block/sda/queue/scheduler
OR
echo deadline > /sys/block/sda/queue/scheduler
cat /sys/block/sda/queue/scheduler

then I ran the script:
Code:

seq 1 10000 | time xargs touch 2> /mnt/usb/tf."$1"
time find 2> /mnt/usb/ff."$1"
time rm -f * 2> /mnt/usb/rf."$1"
seq 1 10000 | time xargs mkdir 2> /mnt/usb/td."$1"
time find 2> /mnt/usb/fd."$1"
time rmdir * 2> /mnt/usb/rd."$1"
time cp /mnt/usb/linux-2.6.39.3.tar.bz2 . 2> /mnt/usb/ct."$1"
time tar -xjf linux-2.6.39.3.tar.bz2 2> /mnt/usb/uk."$1"
time find 2> /mnt/usb/fk."$1"
time rm -rf * 2> /mnt/usb/rk."$1"
time dd if=/dev/zero of=file bs=1M count=1K 2> /mnt/usb/dz."$1"
time cp file file2 2> /mnt/usb/cz."$1"
time cat file > /dev/null 2> /mnt/usb/nz."$1"

So, I got time results for all there commands, then I put them in gnumeric and found the sum of all the times as a general benchmark for the filesystem. If you want, I'll attach the whole thing here, but here are the main results:

Code:

                            total
filesystem        real        user        sys        %CPU        %P        CPU time
cfq:
btrfs                126.6        60.3        51.2        1031.3        88.1        111.5
ext4                145.4        58.7        34.7        910.7        64.2        93.4
jfs                220.2        58.1        31.7        521.5        40.8        89.8
reiserfs        153.4        59.8        54.3        1008.9        74.4        114.0
xfs                482.6        58.6        40.1        425.4        20.5        98.7
deadline:
btrfs                126.3        60.5        51.4        1027.8        88.6        111.9
ext4                139.6        58.4        33.9        919.7        66.2        92.4
jfs                193.0        58.2        31.6        557.2        46.5        89.8
reiserfs        149.8        59.7        54.6        1012.7        76.3        114.3
xfs                485.6        58.7        40.5        428.4        20.4        99.2

According to the time manual I get:

Code:

%P    Percentage of the CPU that this job got, computed as (%U + %S) /  %E.
I was using the built-in time, and as such it didn't have this output option, so I calculated this myself.

I also calculated the CPU time = (%U + %S) = user + system.

What does this all mean ? I know that btrfs is the fastest in real time, and jfs is the fastest in CPU time, but I can't relate the two to make some type of index or meaningful number.

I want to know which of these filesystems is the most efficient, i.e. which one does things in least amount of time and with the least amount of CPU time. How do I integrate these calculations ?

Thanks.

Thor_2.0 09-04-2011 05:08 AM

Okay, so, I'm not really answering but more wondering (willing to learn here, so...) if the cosntellation of forces around the drive is'nt the defining factor as well? Throughput of the controller, bus speed of the motherboard...

Subscribed to this thread, this seems interesting...

Thor

H_TeXMeX_H 09-04-2011 06:00 AM

Yes, it would matter. So far this is best resource I have found on the subject:
http://stackoverflow.com/questions/5...utput-of-time1

Still, it is hard to derive meaningful numbers from this data.

H_TeXMeX_H 09-04-2011 10:16 AM

1 Attachment(s)
I have searched a lot on this, and it looks like the best benchmark in this case is just the real time that each command took.

In that case here are the results in a graph:

Also note that the deadline IO scheduler seems to be better than CFQ overall.

EDIT:

Probably, I will switch to btrfs when they get fsck support and are a bit more stable.

cascade9 09-04-2011 08:22 PM

I'm not sure what you are trying to do with your benchmarking.

I dont know why you would use noatime, its not widely used AFAIK (and it could 'skew' the results), and benchmaking filesystems on a USB flash drive could have _very_ different results to a HDD/SSD.

H_TeXMeX_H 09-05-2011 10:33 AM

I was just saving the results to a USB stick, the benchmarks were done on the HDD.

I was using a netbook for the benchmark, the ASUS Eee PC 1015PN. The HDD is Seagate Momentus 5400.6 series.

I used noatime, because I always use this option. I don't see why it would skew results.

salasi 09-06-2011 06:57 AM

I don't know whether I can answer any of your questions, but I hope that I can add to the discussion, a little.

Quote:

Originally Posted by H_TeXMeX_H (Post 4460429)
I decided to do my own filesystem benchmarking, because I didn't trust Phoronix.

Depending on what exactly you mean, me neither. Here is what I mean:
  • Phoronix is not as good as, eg, gaming benchmarking sites in giving you all of the information that you would need to reproduce their tests.
  • Phoronix has historically tended to go for a 'just do lots of tests and throw it at the wall' testing model; this is as opposed to an 'understand this aspect, interrogate that more closely with that more detailed test' kind of model.
I don't in any way suspect the Phorons of lying or intending to deceive, but I do wonder whether they really take as much advantage of all of the configuration options of more advanced file systems, and maybe their analysis of the data may not be as informative as it might be.

Quote:

Originally Posted by H_TeXMeX_H (Post 4460429)
What does this all mean ? I know that btrfs is the fastest in real time, and jfs is the fastest in CPU time, but I can't relate the two to make some type of index or meaningful number.

I don't think that it is possible to get a single number that would be informative, in all circumstances.

Here is my argument:
  • Imagine a computer starting up; what you want is that the computer starts up as soon as possible, and for processes that have to occur serially (and that may depend somewhat on whether you have been using bootchart to parallelise processes, etc, etc), you just want it to happen in the minimum 'wall time'.
  • Now for the opposite case, you could consider some servers (ones where there is a danger of 'running out of cpu' to complete their work, rather than the more common case, in which waiting for the disks is the bottleneck) you'd really want to minimise the amout of cpu used for a unit of workload.
Those are somewhat synthetic use cases, but you can see that, without at the very least being able to define something about the use cases, you can't just pick out a single number that tells you all about whether 'A' is better than 'B'.
If you can say that you are in one of those corner cases, you can probably say 'this one of the numbers is very much more important than that one; otherwise you have to say that you have to take both into account. Alternatively, you could say, after appropriate research 'in my use case, 70% A and 30% B is my best metric', but you'd always know that there is danger that in some slightly different workload 50;50 (or something else) would be closer.

whatever you do about this, you may not choose the absolute best - there may not be an absolute best, considering different workloads, etc, etc - but you won't be horribly disadvantaged by taking a slightly non-optimal system.

Quote:

Originally Posted by H_TeXMeX_H (Post 4460429)
I want to know which of these filesystems is the most efficient, i.e. which one does things in least amount of time and with the least amount of CPU time

What I'd want would be to get stuff done as soon as is possible, and I don't care much about using a higher amount of cpu to do it. Again, here is my argument
using a computer in interactive mode, I want it to get back to me and let me get on with the next thing as soon as possible; if I've paid for the damn cpu and it uses it, I'm not going to complain about that, doing work is what I bought it for! If I was really, really, pushed, I could maybe get concerned about Watt/hours to complete some task, but I can only really see that as relevant in some embedded task (eg, some media computer) where temperatures might be a concern and it might be operating on battery, otherwise their are better things to worry about
I'd argue that this applies to all GUI and CLI interactive stuff, but if you have a computer doing something extended in batch mode (folding at home, or calculating pi to more digits that seems ever to be useful are the only apps that come to mind, but there are probably others) you might want to take a different attitude.

Quote:

Originally Posted by H_TeXMeX_H (Post 4460429)
What does this all mean ? I know that btrfs is the fastest in real time, and jfs is the fastest in CPU time, but I can't relate the two to make some type of index or meaningful number.

I'd really like to persuade you to use 'wall time' as the preferred term for this; 'real time' is always in danger of being confused with hard/soft real time systems (and 'clock time' is always in danger of causing other confusions); it is not that 'real time' is actually incorrect, it is just that with the other uses of the term, ambiguous (and it can make my head spin decoding which meaning is which, when both are possible).

Quote:

I used noatime, because I always use this option. I don't see why it would skew results.
No, if you have already fixed on the idea that you will use 'noatime' eventually, then, ab definitio, it cannot skew the results. You are benchmarking for a situation in which noatime will always be used and you are doing the benchmarking with noatime. Exactly the right thing to do.

If you are unsure, or some volumes will use noatime and some won't, I still don't expect it to alter the ranking of the results significantly (but I could be wrong, because I haven't done any testing).

BTW, when I did some testing on this, pushing on for 10 years ago, noatime was the biggest single time saver*, so I don't know why people would avoid it (or relatime, maybe). My experience is that, these days, distros often set 'noatime' as the default, which wasn't the case, back then. this is a bit of a disappointment personally, as back then, I had some 'secret sauce' to make a computer go faster, and now I don't, or at least, not to that extent.

Quote:

Probably, I will switch to btrfs when they get fsck support and are a bit more stable.
For me, it is only the fsck that is the show stopper, but I wouldn't be putting it on my laptop or a desktop, so much as a an experimental machine. I'm getting a bit bored with this now, as I thought I'd have been doing this a year ago...

You haven't mentioned journalling modes, and those can cause significant performance differences.

I think there is a general rule 'the more advanced the filesystem, the more knobs you've got to change exactly how they work'. This creates a dilemma as to whether it is fair to, eg, use BTRFS's more advanced options (compression, space cache) in comparison with older filesystems that don't have such options. Until recently, Phoronix has alway dealt with this by just taking 'the default' (and 'the default' can change between different distros and over time, so just saying 'we used the default' doesn't tell you as much as you'd like, and you don't really know if the default is different between, eg, Ubuntu desktop and Ubuntu server), but in their more recent tests, they, at least, have started telling you what options they have taken, which is a step forward.

Next time out, I'll dig out some links, if they are not all broken by now.

* Back then, the second biggest time saver was reiserfs. There are very limited circumstances in which I would even consider using reiser3 today (something that accesses squillions of small files, and for which the individual files are not of the highest value, like the cache files on a squid caching box, maybe), but I might consider reiser4, after appropriate research. But, probably, I'd just go for ext4 or BTRFS, eventually, if I didn't want to do that much work.

cascade9 09-06-2011 06:58 AM

Quote:

Originally Posted by H_TeXMeX_H (Post 4462075)
I was just saving the results to a USB stick, the benchmarks were done on the HDD.

You can tell I'm not a coder/programmer. :|

Reading scripts is something I dont find easy, that (ande speeling) is the only time where my minor dyslexia really shows up.

Quote:

Originally Posted by H_TeXMeX_H (Post 4462075)
I used noatime, because I always use this option. I don't see why it would skew results.

"Do not update inode access times on this file system" might sound fairly innocuous, but who knows quite how it would effect filesystem performance in the real world?

If its a setting you always use, sure, set noatime. But that makes your benchmarking less generalised, to the point where it is of no use to anyone who isnt using noatime (IMO anyway, and I would have to get of my butt and do benchamrking with noatime set and compare it to 'standard' settings to be able to say that for sure).

Torvalds has had a rant about noatime...like most things. ;)

http://kerneltrap.org/node/14148

Quote:

Originally Posted by salasi (Post 4462797)
Depending on what exactly you mean, me neither. Here is what I mean:
  • Phoronix is not as good as, eg, gaming benchmarking sites in giving you all of the information that you would need to reproduce their tests.
  • Phoronix has historically tended to go for a 'just do lots of tests and throw it at the wall' testing model; this is as opposed to an 'understand this aspect, interrogate that more closely with that more detailed test' kind of model.
I don't in any way suspect the Phorons of lying or intending to deceive, but I do wonder whether they really take as much advantage of all of the configuration options of more advanced file systems, and maybe their analysis of the data may not be as informative as it might be.

I see your point, and I agree that you're right in some ways...the thing is that phoronix is walking a fine line between the far more detailed but dry and difficult to read analysis you can find on other sites, 'I just want to know what runs faster without mucking around with options'.

I'm far more wary of 'gaming' benchmarking myself. Sometimes its worth it, sometimes the analysis is out and out illogical.

Far more worring is that there are a metric butt-load of sites with 'gaming benchmarking' that has been cherry picked to make system 'A' look better than system 'B'.

*edit- I'm not even that convinced by the whole 'time' issue. Heres my reasoning (and I've never done any benchmarking on this exact situation)

Filesystem 'A' takes 30 seconds to write file 'A' and finalise. CPU and I/O use is high for 5 seconds as it writes the file, then drops off for the next 24 seconds, then there is one spike in CPU and I/O use as it finalises the writing.

Filesystem 'B' takes 20 seconds to write file 'A' and finalise. CPU and I/O use is high for the whole 25 seconds.

Which one is better?

If your only concerned about CPU use, 'A' is better. If your concerned about disc I/O, 'A' wins again. But in total time 'B' is better.

salasi 09-06-2011 07:45 AM

Quote:

Originally Posted by cascade9 (Post 4462798)

"Do not update inode access times on this file system" might sound fairly innocuous, but who knows quite how it would effect filesystem performance in the real world?

Me. I did the testing. As I say, it is nearly ten years ago, so it may take a while to dig out the results and they'd be of limited relevance today (minimal cpu available on a single-core 366MHz pentium, but the hard disk will have been slower, too).

But the bottom line is that it is clear. Back in the day, I used to claim that I could tell by just the responsiveness of the system whether atime was on or off, without even knowing the processor speed, looking at the fstab setting, etc. It is probably harder, these days, as there probably a bigger spread of levels of hardware performance, but it is that clear, it does make a noticeable difference.

There is one factor that hasn't been mentioned so far; the number of cores. If you have several cores, who cares that you saturate one with the disk stuff, you've got enough others? So, the best solution for a monocore machine may well not be the best solution for a many-core box.


Quote:

Torvalds has had a rant about noatime...like most things. ;)

http://kerneltrap.org/node/14148

In many ways, its is good to know that the world still turns, as usual, and Linus can be relied upon to speak his mind.

Quote:

Originally Posted by cascade9 (Post 4462798)
I see your point, and I agree that you're right in some ways...the thing is that phoronix is walking a fine line between the far more detailed but dry and difficult to read analysis you can find on other sites, 'I just want to know what runs faster without mucking around with options'.

I'm far more wary of 'gaming' benchmarking myself. Sometimes its worth it, sometimes the analysis is out and out illogical.

Far more worring is that there are a metric butt-load of sites with 'gaming benchmarking' that has been cherry picked to make system 'a' look better than system 'b'.

That fine line isn't necessarily a big thing. 'Best practice' (which I've seen defined as 'what you can get away with') is to put all the dull and boring set-up stuff on one page, that is easy to ignore, if you want to. (Why they don't make this the last page of a multi-page review, I don't know; that would make it really easy to ignore, if you aren't interested.)

There are indeed a lot of 'fanboi' sites out there in the gaming world, and you will note that I didn't suggest that they should be followed slavishly, just that they did better, in this one aspect. And even the non-fanboi gaming sites get deluged with hate mail from fanbois of one team or the other wanting to scream about the fact that their preferred system didn't get praised to the roof in the latest testing, claiming that only a fanboi of the opposing team could have done this. Ho, hum.

cascade9 09-06-2011 08:29 AM

Quote:

Originally Posted by salasi (Post 4462827)
Me. I did the testing. As I say, it is nearly ten years ago, so it may take a while to dig out the results and they'd be of limited relevance today (minimal cpu available on a single-core 366MHz pentium, but the hard disk will have been slower, too).

I guess you used Ext3 vs reiserfs, maybe others? I dont know how that would translate to more modern systems (like you said). It would be interesting to see though.

Quote:

Originally Posted by salasi (Post 4462827)
There is one factor that hasn't been mentioned so far; the number of cores. If you have several cores, who cares that you saturate one with the disk stuff, you've got enough others? So, the best solution for a monocore machine may well not be the best solution for a many-core box.

So_many_variables. Which is why I agree 100% with the idea that there is no 'single best answer'.

Quote:

Originally Posted by salasi (Post 4462827)
That fine line isn't necessarily a big thing. 'Best practice' (which I've seen defined as 'what you can get away with') is to put all the dull and boring set-up stuff on one page, that is easy to ignore, if you want to. (Why they don't make this the last page of a multi-page review, I don't know; that would make it really easy to ignore, if you aren't interested.)

There are indeed a lot of 'fanboi' sites out there in the gaming world, and you will note that I didn't suggest that they should be followed slavishly, just that they did better, in this one aspect. And even the non-fanboi gaming sites get deluged with hate mail from fanbois of one team or the other wanting to scream about the fact that their preferred system didn't get praised to the roof in the latest testing, claiming that only a fanboi of the opposing team could have done this. Ho, hum.

Its not just the 'fanboi' sites, IMO a lot of gaming benchmarks are just as much 'just do lots of tests and throw it at the wall' as phononix, if not more so. Of course there are some very good sites with analysis on gaming benchmarks. Considering that gaming benchmarking is more common than other benchmarking I'd expect that some places at least would get it right.

I dont think that its really a fair comapison with phoronix, there are only so many variables with windows. You've got more variables with linux distros.

Most of the time when I've looked at phononix I've seen all info needed. Yeah, they tend to just use 'defualt' values for filesystems with whatever distro they are using for the test, but IMO thats no great issue. Most people just use the defaults.

H_TeXMeX_H 09-06-2011 12:07 PM

Thanks for the info and links.

The first thing I tried for this benchmark was phoronix test suite, but it was hard to use, and the disk benchmarks didn't seem close enough to real life to be accurate. I don't use SQlite, for example. I don't how the other benchmarks work, so I decided to just do some real life tests. I based them on:
http://linuxgazette.net/122/piszcz.html
even tho the author never released the commands used or more details.

As for noatime, I would have to agree with Linus T and Ingo Molnar, that it is definitely noticeable, and probably the best single option that will speed up your disk access.

salasi 09-06-2011 01:38 PM

first, i did promise some links; these are from Linux Magazine; the guy who wrote these seems to have had a particular thing about metadata performance and also about doing benchmarking correctly, so these are worth reading, if just for the ride. OTOH, I think you have to register to see the actual articles, if you haven't already. (If you are serious about this, these articles are worth reading, whether you are going to end up agreeing with them, or not - I'm not normally much of a fan of unnecessary registration, but it is not that painful.)

The whole tone of the article is less 'pop' than some of the Phoronix ones, but that doesn't necessarily make it better or worse.

Lies, Damn Lies and File System Benchmarks
Harping on Metadata Performance: New Benchmarks
Metadata Performance of Four Linux File Systems
Metadata Performance Exploration Part 2: XFS, JFS, ReiserFS, ext2, and Reiser4


Quote:

Originally Posted by cascade9 (Post 4462871)

Most of the time when I've looked at phononix I've seen all info needed. Yeah, they tend to just use 'defualt' values for filesystems with whatever distro they are using for the test, but IMO thats no great issue. Most people just use the defaults.


Well, a lot of the recent tests are done on a laptop, which defines things to an extent, but do you know what memory timings they run, whther that's the standard hard disk (and whether that laptop manufacturer multi-sourced its hard disks, so whether there is only one option in that size) and they could have even overclocked it. They probably haven't, but a list wouldn't be a bad thing, would it?

H_TeXMeX_H 09-06-2011 02:25 PM

Thanks again, those are very good articles. I did have to register tho, even tho I don't like to.

EDIT:
btrfs still looks like the winner even from these.

catkin 07-16-2013 02:58 AM

Quote:

Originally Posted by H_TeXMeX_H (Post 4460429)
...
then I ran the script:
Code:

seq 1 10000 | time xargs touch 2> /mnt/usb/tf."$1"
time find 2> /mnt/usb/ff."$1"
time rm -f * 2> /mnt/usb/rf."$1"
seq 1 10000 | time xargs mkdir 2> /mnt/usb/td."$1"
time find 2> /mnt/usb/fd."$1"
time rmdir * 2> /mnt/usb/rd."$1"
time cp /mnt/usb/linux-2.6.39.3.tar.bz2 . 2> /mnt/usb/ct."$1"
time tar -xjf linux-2.6.39.3.tar.bz2 2> /mnt/usb/uk."$1"
time find 2> /mnt/usb/fk."$1"
time rm -rf * 2> /mnt/usb/rk."$1"
time dd if=/dev/zero of=file bs=1M count=1K 2> /mnt/usb/dz."$1"
time cp file file2 2> /mnt/usb/cz."$1"
time cat file > /dev/null 2> /mnt/usb/nz."$1"

...

Hi H_TeXMeX_H :) What's the $1 in this script?

H_TeXMeX_H 07-16-2013 03:39 AM

You pass in the name of the filesystem, that way you can tell what filesystem the test was performed on.


All times are GMT -5. The time now is 10:02 AM.