LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Server (https://www.linuxquestions.org/questions/linux-server-73/)
-   -   Best filesystem for my Linux datastore server: XFS or EXT4? (https://www.linuxquestions.org/questions/linux-server-73/best-filesystem-for-my-linux-datastore-server-xfs-or-ext4-4175616285/)

gda 10-24-2017 09:24 AM

Best filesystem for my Linux datastore server: XFS or EXT4?
 
Hi all!

I'm trying to design a linux datastore server and I'm facing out with the question of which filesystem type would meet better my actual needs.

Few words about the requirements:

- Volume size 40TB (RAID 6)
- The volume is expected to be populated mainly by two different subsets of datafiles: one with size around 100-200MB and another one with many files of size less than 1MB
- I need to read and write as fast as possible both these file subsets
- Finally I would use a solution based on a stable and mature filesystem which can be extended without data loss if necessary in future.

According to what I have found around it seems two filesystem types meet quite well all the requirements: EXT4 or XFS.

From what I have found both these filesystems have advantages and disadvantages. For example, what I like about XFS is that it doesn't need to run tools like e2fsck regularly as EXT4 does which I think is a good point especially if we are talking about large volumes. Moreover, XFS is really mature and so it is supposed to be very stable. On the other side, I have read several cases complain about not really high performances of XFS and troubles in recovering the data for example after a power failure. I know that a power failure can be critic for any filesystem (that's why I have an UPS in my infrastructure) but from what I have understood this is particularly critic for XFS.
The EXT4 seems to have better performances but it is not mature as the XFS. For sure the EXT4 is much more used with respect to the XFS so I suppose linux kernels and maybe specific applications can be somehow more optimized for an EXT4 rather then XFS. Personally I have no experience in running EXT4 on volume larger than 2TB... I have a couple of large volumes (on another infrastructure) running on the top of XFS but they are pure archive where fast read and write are not so important (by the way they are up since 2011 without any problem!).

To conclude I'm a bit confused about which would be the best choice for my case.

Any suggestion or recommendation is really appreciated!

Thanks a lot for your attention.

Best regards.

MensaWater 10-24-2017 09:48 AM

ext4 is an extension of the original ext2 then ext3 filesystems. ext4 itself has been available and in widespread use for several years and given it is an extension of those earlier filesystems I'm not sure why you think it isn't "mature".

You do NOT have to run e2fsck on ext4 frequently.
It does run on reboot usually since it is a journaled filesystem the fsck usually just needs to verify there are no uncommited writes in the journal. It will apply those if found but that would usually only happen in the event of a sudden power loss to the server.

You can disable automatic fsck for ext4 if you want though I generally don't since it isn't slow to run. (ext3 on the other hand was dog slow because it did a full fsck every time rather than just looking for uncommitted writes in the journal. ext was not journaled at all so always had to do a full fsck.)

ext4 unlike xfs can be reduced (if done carefully with planning) whereas xfs can never be reduced.

gda 10-24-2017 12:15 PM

Dear MensaWater,

thanks a lot for your replay.

Quote:

Originally Posted by MensaWater (Post 5773332)
ext4 is an extension of the original ext2 then ext3 filesystems. ext4 itself has been available and in widespread use for several years and given it is an extension of those earlier filesystems I'm not sure why you think it isn't "mature".

Officially EXT4 has been released on 2008. So I supposed it can be considered a relatively young filesystem with respect to XFS which has been released much earlier. It is true it is an extension of ext2 and ext3 but actually it implements completely new stuff to speed up the performances with respect to EXT3.

Quote:

Originally Posted by MensaWater
You do NOT have to run e2fsck on ext4 frequently.
It does run on reboot usually since it is a journaled filesystem the fsck usually just needs to verify there are no uncommited writes in the journal. It will apply those if found but that would usually only happen in the event of a sudden power loss to the server.

You can disable automatic fsck for ext4 if you want though I generally don't since it isn't slow to run. (ext3 on the other hand was dog slow because it did a full fsck every time rather than just looking for uncommitted writes in the journal. ext was not journaled at all so always had to do a full fsck.)

Sorry my fault... I was not clear enough on that... What I wanted to say is that it is recommended to run fsck at every reboot and you said and it is also recommended to run e2fsck when a certain configurable maximum mount counts or check time is reached.
As I said I have no experience in running such tools on large EXT4 volumes. Do you have experience on that? How long approximately it will take to run fsk or e2fsck on an EXT4 volume of about 40TB in case it is found clean?

Quote:

Originally Posted by MensaWater
ext4 unlike xfs can be reduced (if done carefully with planning) whereas xfs can never be reduced.

Correct! I forgot to mention that.

Anyway according to my requirements, I understand you would go for EXT4 instead of XFS. Is that correct? Could you provide me more details on that?

Thanks again!

MensaWater 10-24-2017 01:13 PM

9 year old still supported software would classify as "mature" in my view even if it didn't have the history behind it. In point of fact ext4 today is likely different than it was in 2008 as I expect xfs today is different than when it was first released.

I have no problem with xfs but so far haven't seen a need to go that way on anything.

Our largest filesystem on ext4 currently is just over 7 TB. We have multiple of those.

The longest time I see running fsck (e2sfsck) is on the server where we have literally dozens of filesystems. The time there is in processing the number of filesystems as opposed to the size. Even there we're talking minutes rather than hours. (On ext3 we often saw extremely long times for large filesystems.)

I'm not suggesting ext4 is better (or worse) than xfs - I was just challenging some of what you wrote. What I would pick would depend a lot on capabilities and expected growth. I haven't checked recently but I believe xfs can grow larger than ext4. So far I've not had a need that made want to go away from ext4.

gda 10-24-2017 03:13 PM

Really thanks for the further clarifications and to have reported your experience. Mainly this is the information I was looking for.

Yes, theoretically XFS performs better than EXT4 in terms of maximum volume size and maximum file size. Anyway we are talking about differences in sizes which probably I will never notice on the system I'm trying to set up...

I'm more interested in stability and performance. I could compare read/write performances of EXT4 and XFS relatively easily (I have already planned to do something like that and if I have time I will report the results here) but I don't know any easy way to compare the stability... I think in this case only long time experience on filesystems (which unfortunately I don't have) can provide reliable answers...

MensaWater 10-25-2017 07:47 AM

We've been using ext4 pretty extensively for several years. Long ago we converted our ext3 filesystems to ext4 once RHEL "officially" supported it on RHEL5. RHEL6 has ext4 as the default and we've been using it there since inception. RHEL7 also has ext4 as a default and also has xfs available. On some of the early RHEL7 we used xfs as default but only for the base system (/, /usr, /var, /tmp etc...). For new application and database filesystems we always use ext4. Since CentOS is a binary compile from RHEL sources this is true for CentOS 7. We have a large Couchbase distributed DB on multiple CentOS7 servers using ext4.

We also use CoreOS on multiple servers on which we are running Docker containers for the past couple of years. The default FS for CoreOS itself is also ext4 though it uses a fair number of tempfs mounts for the various containers one puts on it.

In summary I feel safe in saying ext4 is "stable".

gda 10-27-2017 04:27 AM

Thanks once more for the detailed report... I really appreciate you took some of your time to write it down...

Here I'm with some comparisons about the performance of XFS and EXT4 I/O performances.

I have performed all the testing using a VM with the following characteristics:

CPU: 4 cores
RAM: 8GB
OS: Slackware64 14.2

On this VM it is mapped a raw SAN volume (FC) of 40TB.

First I have formatted this raw volume using one single XFS partition and I have performed some I/O tests on it(using tools like dd, hdparm,...). Then I have re-formatted the same raw volume using one single EXT4 partition and I have repeated the same tests I previously ran on the XFS partition.

This is summary of the results I got:
Code:

Write Speed [MB/s] (using dd)
Packet size  EXT4  XFS        XFS improvement
10MB          320    333        4.06% 
1KBx10240    0.473  0.73      54.33%
1GB          315    326        3.49% 
1MBx1024      127    143        12.6%
1GBx10        292    313        7.19% 
1GBx100      299    323        8.03% 

Read Speed [MB/s] (using dd)
Packet size  EXT4  XFS        XFS improvement
10MB          511    433        -15.26%
1GB          618    443        -28.32%
10GB          618    444        -28.16%
100GB        621    569        -8.37%         
                                               
Cached Reads [MB/s] (using hparm)                                                     
EXT4    XFS        XFS improvement     
7761.47 7822.61    0.79%                               
                                                               
Buffered Reads [MB/s] (using hparm)                   
EXT4    XFS        XFS improvement     
570.59  479.67    -15.94%                                             
                                               
Latency [ms]                                           
EXT4      XFS        XFS improvement   
0.226793  0.16867    25.63%

I have to say that these results look a bit surprising to me... I was quite confident to find better overall performance of EXT4 but actually I got something different: it seems the XFS performs better in writing while EXT4 performs better in reading. Moreover, in general it seems that as the packet size increases the performance of the two filesystems tends to be similar (also in this case I was expecting to find an increasing of the performance of XFS with respect to EXT4).

I have also done some writing tests using different concurrent threads and I found that as the number or threads increases the EXT4 and XFS show similar performance (which is again something I did not expect!)

I know the tests I have made are really basic but actually they provide already a quite clear result about the performance. What do you think? Did I miss something? Do you have suggestions on how to check the performances of the two filesystems in a more reliable way? Any comment is really welcome!

Greetings

gda 10-27-2017 06:22 AM

I have repeated the same tests described above but disabling both read and write cache on the SAN (I think this make more sense for the tests I'm doing...).

These are the results I got:

Code:

Write Speed [MB/s] (using dd)
Packet size  EXT4  XFS        XFS improvement
10MB              111    301        171.17%  (!) I have repeated serveral times this test and the result are always similar
1KBx10240    0.165  0.392        137.58%
1GB              371    393        5.93%
1MBx1024      102    176        72.55%
1GBx10              394    414        5.08%
1GBx100              395    411        4.05%

Read Speed [MB/s] (using dd)
Packet size  EXT4  XFS        XFS improvement
10MB          587    634        8.01%
1GB          651    652        0.15%
10GB          649    653        0.62%
100GB        607    652        7.41%
                                               
Cached Reads [MB/s] (using hparm)                                                     
EXT4    XFS        XFS improvement
7695.67        7736.86          0.54%
                                                               
Buffered Reads [MB/s] (using hparm)                   
EXT4    XFS        XFS improvement     
447.12        613.05          37.11%
                                               
Latency [ms]                                           
EXT4      XFS        XFS improvement   
0.568047  0.311678  45.13%

As you can see the situation seems to be even more shifted toward the XFS... For same reason without caching XFS seems to improve reading speed with respect the configuration in which the caching is enable. I really don't know why this happens...

gda 10-31-2017 03:27 PM

Any comments about my tests? Do my results make any sense to you?

IsaacKuo 10-31-2017 05:21 PM

I'm just surprised that none of your tests targeted the file sizes you said you were expecting (100-200MB, and something less than 1MB).

gda 10-31-2017 05:55 PM

Actually I did further tests using datasizes more close to the expected ones but I didn't mention them because the results were similar to the ones already shown above: XFS seems to perform better then EXT4 also in that case.
As I didn't expect to see such a clear supremacy of XFS with respect to EXT4 I'm wondering if I miss something or, even worse, I did something wrong in my tests. On Internet I found similar tests but with different results... I'm a bit confused...

jefro 10-31-2017 09:43 PM

Part of this problem is two fold. The kernel version you may be using and the hardware could greatly affect your tests. Generally I look to the tests online and try to look for a few kernel versions before the one I am using to decide what I might expect.
The online web pages use a number of tests that may be of interest to your choice other than a dd test.
You also have a few more choices out there if you are just looking for speed.



Side note.

Ext4 may be more supported in various tools and recovery means. XFS may demand more backing up because of less support.

gda 11-01-2017 05:13 AM

Thanks for the reply!

You are completely right in saying there are other tools dd. As I wrote above, along with tools like dd and hdparm (mainly single thread sequential read/write tools) I have used also multithread random write tools (like the one described here) obtaining similar results.

Concerning the kernel I can imagine that this is an important parameter to take into account. As I said I have used Slackware64 14.2 which runs a vanilla kernel 4.4.88. As this kernel is not customized I suppose there should be no important kernel-specific reasons to explain what I found. Is that correct?

Thanks again for the help!

IsaacKuo 11-01-2017 10:00 AM

Quote:

Originally Posted by gda (Post 5775742)
Actually I did further tests using datasizes more close to the expected ones but I didn't mention them because the results were similar to the ones already shown above: XFS seems to perform better then EXT4 also in that case.
As I didn't expect to see such a clear supremacy of XFS with respect to EXT4 I'm wondering if I miss something or, even worse, I did something wrong in my tests. On Internet I found similar tests but with different results... I'm a bit confused...

So, XFS performed better with the caches turned off, while EXT4 performed better with the caches turned on?

I dunno...it seems like you really care about minor performance differences, but your decision to use RAID6 instead of some form of RAID1+RAID0 seems to matter more than XFS vs EXT4. With the caches turned on, it seems the performance difference is not much. But RAID6 vs RAID1/RAID0? That could be a really big difference.

The performance hit will be particularly bad if a drive fails. With some combo of RAID1/RAID0, the performance hit will be minimal even after the failed drive is replaced. With RAID6, you're looking at the potential for a major drag on performance for a long time. With a RAID1/RAID0 scheme, you're looking at a small performance hit for a pretty short time.

If I were in your position, I'd just stick with XFS out of familiarity. Since you seem to only care about expanding and not shrinking, ext4 lacks a "killer feature" to compel a switch. (IMHO, any situation where you're trying to recover data because it wasn't backed up elsewhere is...eh, no thanks. Make sure the backups are good and have at least two good independent backups for everything that matters!)

As for achieving the desired performance levels...I'd look at the RAID scheme and probably do some performance testing - split up the drives into a 20TB RAID6 and a RAID1+0 with the other half of space.

gda 11-01-2017 02:47 PM

Dear IsaacKuo,
thanks a lot for your suggestions.
You are completely right in saying that I would gain much more in changing the RAID type then in changing the filesystem type.
Anyway unfortunately I cannot go for a RAID10 because I have a strict constraint on the volume size which should be at least 40TB. With my current disks setup I can archive that only using a RAID6 based solution. So I'm just trying to get the best I can get from the option I'm forced to go...

I fully agree with your considerations about a solid backup strategy... I would never fully trust in the recovery option of any filesystem type especially if we are talking about important data...
That's why in my setup I have a second (less performance) SAN which is dedicated only to backup purposes.

Having said that I would go for the XFS... The residual doubts I still have are mainly triggered by some cases of bad experience in using XFS I found googling on internet... Not sure if in all these cases the real and unique problem was the XFS...

So it would be really useful for me to know about the experience (positive or negative) of anybody here with XFS.

Thanks once for more for the nice support I'm getting here!


All times are GMT -5. The time now is 10:43 AM.