FAT32 R/W performance too low for big File transfer

chinmoy · 01-13-2019, 01:12 PM

i have mounted FAT32 formatted USB storage to etx4 FS and executed large file copy operation . Observed that FAT32 R/W performance to ext4 partition is very much slower than same ext4 partitions .

I've tried various combinations of with mount with ASYNC option and increased the FAT cluster but it didn't help though.

Debugged the v4.4.84 kernel and looks like following funcs. are taking more time .

Code:

fat_write_end(){ 
   ...
      block_write_end(){
        ... 
        grab_page_cache_write_begin(){
            ...

        }   //+96.923us

    }   //+168.089us

 }  //+189.778us

Could you please suggest how to improve the performance.

Help is greatly appreciated!

BW-userx · 01-13-2019, 01:23 PM

FAT32 hum 32 -- hum, how old is FAT32 and when was the last time they updated that method of file storage? just ball parking it here, but that is an old DOS format that was something to behold when it went from FAT to FAT32. I am sure there are write and read limitations on it as all formats. I'd think ext4 being a standard format in Linux it would be kept updated, even in its performance not only for how it stores data, but how it reads and writes it to a partition.

Whereas FAT32 is just something that is still in the mix because it is hard to just get rid of it, for whatever the reasons. You might even be a perfect example because you're still using FAT32 for whatever your reasons.

I am sure there are tech heads in here that can give you a much better techy details on this.

ehartman · 01-13-2019, 02:16 PM

Quote:

Originally Posted by BW-userx

FAT32 hum 32 -- hum, how old is FAT32 and when was the last time they updated that method of file storage?

Fat32 was introduced in one of the update releases for Windows-95 and before Windows-98 came, about 1996, so more then 20 years ago.
Not too long ago a further extension/update was released by M$, called exFAT

BW-userx · 01-13-2019, 03:17 PM

Quote:

Originally Posted by ehartman

Fat32 was introduced in one of the update releases for Windows-95 and before Windows-98 came, about 1996, so more then 20 years ago.
Not too long ago a further extension/update was released by M$, called exFAT

see I knew a techy would show up.

me just back yard logic on this, FAT32 could only read and write x-amont of data on one time, no matter if it is being written to some type of format that supersedes it. Basic Bottle necking is going to occur.

https://support.microsoft.com/en-us/...-fat-and-exfat

performance
https://www.flexense.com/fat32_exfat...omparison.html

then one needs to take into consideration through put mediums, data being transferred on the same hard drive as apposed to one hard drive to the other, along with the means used to connect the two.

same drive all that needs to be done it a quick rewrite in the file listings to where it is "now located" within the file system, between two drives actual moving of the data has to occur and be written to where it is located on the file listings in the other drive, and removed from the original drive, which takes more time.

vivekvikky · 01-14-2019, 12:52 AM

Even i have similar issue when i have executed iozone to benchmark different file systems some time back. I have run iozone test which took around 10 mins to complete whole set of iozone operations on ext4 . where took approximately ~2 hours to complete whole iozone set (read/write/ rewrite / reread/ fread/ fwrite ...) on FAT32 . Both operations were performed on same disk with different partitions.

ondoho · 01-14-2019, 01:38 AM

maybe the bottleneck is the USB connection, not the filesystem?

chinmoy, how are you connecting the external storage? usb 2 or 3 or, god beware, 1.1?

also it stands to reason that even usb 3 transfer speeds will always be slower than internal copying on the same filesystem.

chinmoy · 01-14-2019, 06:09 AM

Hi ondoho,
I have tried with USB 2.0 only .

wpeckham · 01-14-2019, 06:55 AM

Quote:

Originally Posted by chinmoy

Hi ondoho,
I have tried with USB 2.0 only .

Well USB2 is much slower than the internal drive interface speeds. USB3 is slower than many, but is far closer and faster than a couple older storage interfaces (ie. much better if it is an option).

While FAT32 is slower than EXT4, many inexpensive USB devices do not react at the speed of the USB interface. I know this because I tend to use mostly the most inexpensive USB devices and run into it a LOT.

If you have a second identical USB device formatted to EXT4 you might want to perform some quick data copy tests to verify that the file system is actually involved in this case. The results would be specific to your exact hardware combination, but would help you eliminate or confirm one factor in this equation.

If you perform such a test, please share your results. I would really like to see what obtains.

chinmoy · 01-14-2019, 07:19 AM

Yeah .I have already tired and formatted with EXT4 fs and seen the better performance than EXT4 .
Even i have tried with Approach 3 configuration of https://lonesysadmin.net/2013/12/22/...m-dirty_ratio/ link and found the strange behavior.
Executed two test cases :
Case 1 : copied a 256 MB file with bs of 64K
Case 2 : Ran iozone whole test suit

Observation :
Case 1 is giving better result but IOZONE performance (Case 2) is worst than default page cache configuration .