LinuxQuestions.org
Help answer threads with 0 replies.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 04-28-2022, 07:19 AM   #1
DrDwayne
Member
 
Registered: May 2010
Location: Wichita
Distribution: Ubuntu Debian
Posts: 155

Rep: Reputation: 4
Exfat corrupting large thumb drives.


I have been having this problem for quite some time.
Scenario. . .
Ubuntu 21. Has happened to any version of Linux I use.
Thumbdrive = 512Gig (or 128G. 64gig)
ExFat installed on Linux with no problems.

At first the Drive seems ok when I copy. But if I do a
huge copy of files, (let's say 10 gig or so). It falsely
tells me the files are copied, space is "possibly" taken, but no files
are copied.

I have noticed that the Directories I created DO Exist, but the
files I copied into them (anywhere between 100meg to 500meg) do
not exist at all. I have wasted hours of my time for nothing.

I cannot dismount and remount in hopes to recover the files,it doesn't work.
I go to a windows computer and it says thumbdrive corrupt. So I fix it, and
the files do not exist.

This seems to only happen in the ExFat formatted thumb drives.
Is there a fix for this problem? IT is such a problem, that I can almost
guarantee a corrupt thumbdrive if I use one in Exfat.
 
Old 04-28-2022, 08:49 AM   #2
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 22,039

Rep: Reputation: 7347Reputation: 7347Reputation: 7347Reputation: 7347Reputation: 7347Reputation: 7347Reputation: 7347Reputation: 7347Reputation: 7347Reputation: 7347Reputation: 7347
need to know how did you do that exactly and how did you check exactly. What I think is:
you execute the copy to usb, but did not wait for completion. You need to [safely] eject and/or umount your usb stick, cannot be removed without that (that's why you get this error).
 
Old 04-28-2022, 09:48 AM   #3
boughtonp
Senior Member
 
Registered: Feb 2007
Location: UK
Distribution: Debian
Posts: 3,627

Rep: Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556

To expand on what Pan says...

Quote:
Originally Posted by DrDwayne View Post
It falsely tells me the files are copied
Sometimes there is a false message that a copy has completed when the files are only in the cache, not yet persisted to disk.

If you're in a terminal, the sync command can be used to ensure the cache is written - run it in a terminal and wait for it to come back (for 10GB of data it may take a while).

This is what a GUI "safely remove" (or similarly named option) should also do, and why you should always use such options before removing a storage device (whether on Windows or Ubuntu or any other OS).

 
Old 04-28-2022, 10:43 AM   #4
DrDwayne
Member
 
Registered: May 2010
Location: Wichita
Distribution: Ubuntu Debian
Posts: 155

Original Poster
Rep: Reputation: 4
It doesn't matter how long I leave the USB drive in,
whether I choose to "Safely Eject" and wait for the OS to say "It is now safe to remove XXX "

Anything to do with Exfat is almost a guaranteed failure. . .

With that said. . .

I can read without seemingly any problems at all, I just can't copy things
to the thumb drive with any guarantee.

It can be pictures, it can be movies, it can be text files.

I have tried probably 5 different brands of thumb drives. . .from A to Z.

I have zero problems copying large files to thumb drives that are 16gig or less, Even 32gig.
They all work perfectly.
 
Old 04-28-2022, 11:14 AM   #5
boughtonp
Senior Member
 
Registered: Feb 2007
Location: UK
Distribution: Debian
Posts: 3,627

Rep: Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556

Ok, so you've now revealed a whole bunch of new information that should have been in the first post!

There's still important details missing though...

* "Ubuntu 21" could refer to either 21.04 or 21.10 - which one?
* The kernel version you're running is relevant to what sort of exFAT support you have - what version of Linux is it?
* Does "ExFat installed on Linux" mean you manually installed the old non-kernel drivers that your OS probably doesn't need?

* You've not said if the devices/ports are USB 2 or USB 3
* Did they come pre-formatted as exFAT or did you format them that way (if so, how?)
* What precisely are those "5 different brands"?

You can run "inxi -SMxxz --usb" to provide some of that - post the results inside "[code]..[/code]" tags.

 
Old 04-28-2022, 11:16 AM   #6
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 7,357
Blog Entries: 3

Rep: Reputation: 3767Reputation: 3767Reputation: 3767Reputation: 3767Reputation: 3767Reputation: 3767Reputation: 3767Reputation: 3767Reputation: 3767Reputation: 3767Reputation: 3767
It should be a technical problem but it is not. It is a political problem.

Both the politics and the data loss are in the nature of (Ex)FAT. It was designed for 5.25" floppies and is most unsuitable for anything of modern size as the design basically guarantees that it will lose your data. Try a different format if you can. The catch is if you can, since M$ has multiple software patents encumbering ExFAT in the US it has financial incentive to leverage its position to exclude all other formats, no matter how much more appropriate they may be for storage media.

Maybe that information is not much help beyond providing context for the problem.
 
Old 04-28-2022, 11:30 AM   #7
DrDwayne
Member
 
Registered: May 2010
Location: Wichita
Distribution: Ubuntu Debian
Posts: 155

Original Poster
Rep: Reputation: 4
One thing that I can be sure of. . .to a degree. . .
If I copy just one directory of files at a time, the success rate is extremely high.
That I remember, I don't think it has failed.

If I copy multiple directories at the same time with many files,
the failure rate is extremely high. I would say almost 100 percent.
 
Old 04-28-2022, 11:32 AM   #8
michaelk
Moderator
 
Registered: Aug 2002
Posts: 25,783

Rep: Reputation: 5936Reputation: 5936Reputation: 5936Reputation: 5936Reputation: 5936Reputation: 5936Reputation: 5936Reputation: 5936Reputation: 5936Reputation: 5936Reputation: 5936
Quote:
Both the politics and the data loss are in the nature of (Ex)FAT. It was designed for 5.25" floppies and is most unsuitable for anything of modern size as the design basically guarantees that it will lose your data.
?????

exFAT was released in 2006 AFAIK designed for embedded systems like cameras with memory cards to overcome the 4GB FAT32 limitation.

I agree it might be due to a conflict if both fuse and the kernel module are loaded at the same time.
 
Old 04-28-2022, 03:55 PM   #9
jefro
Moderator
 
Registered: Mar 2008
Posts: 22,020

Rep: Reputation: 3630Reputation: 3630Reputation: 3630Reputation: 3630Reputation: 3630Reputation: 3630Reputation: 3630Reputation: 3630Reputation: 3630Reputation: 3630Reputation: 3630
Kind of odd that exfat fails on a major distro like that. I'd be more inclined to believe that the usb controller timing is at fault. Are you using an add on usb board?

Otherwise if you can try this on a different computer it might prove the usb drive or computer at fault.

My guess is that it isn't exfat filesystem. That doesn't mean that all the parts going to and from it are good.
 
Old 04-28-2022, 04:59 PM   #10
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,152

Rep: Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125Reputation: 4125
I rarely use exfat, only for big data where I can't/won't use NTFS - never had an issue that I can recall.

Not so long ago we had a user rant here on LQ about never to use FAT - of any description. Had a quick look but couldn't find the thread.
 
Old 04-29-2022, 12:23 AM   #11
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 22,039

Rep: Reputation: 7347Reputation: 7347Reputation: 7347Reputation: 7347Reputation: 7347Reputation: 7347Reputation: 7347Reputation: 7347Reputation: 7347Reputation: 7347Reputation: 7347
Quote:
Originally Posted by DrDwayne View Post
One thing that I can be sure of. . .to a degree. . .
If I copy just one directory of files at a time, the success rate is extremely high.
That I remember, I don't think it has failed.

If I copy multiple directories at the same time with many files,
the failure rate is extremely high. I would say almost 100 percent.
you can still try to use sync, fsck and similar tools to check that device. And also you can try another filesystem if that works at all on that usb stick.
 
Old 04-30-2022, 11:19 AM   #12
DrDwayne
Member
 
Registered: May 2010
Location: Wichita
Distribution: Ubuntu Debian
Posts: 155

Original Poster
Rep: Reputation: 4
Quote:
Originally Posted by jefro View Post
Kind of odd that exfat fails on a major distro like that. I'd be more inclined to believe that the usb controller timing is at fault. Are you using an add on usb board?

Otherwise if you can try this on a different computer it might prove the usb drive or computer at fault.

My guess is that it isn't exfat filesystem. That doesn't mean that all the parts going to and from it are good.
Every distrobution I have ever ran that accepts large thumbdrives has had this issue. . .on any computer.

I can read from the thumb drive without anyproblem with the ExFat drivers installed. but when writing to
the thumbdrive. . .there is no guarantee. . .

I am not sure, but I have noticed that if I have multiple files writing to the
thumb drive at the same time, it seems to fail on me.
It shows everything is copying, but when all said and done, everything I did is completely
lost.

I stick the drive into a windows unit, and everytime it says "Thnumdrive has errors on it, do you want to
scan and fix".

Then i get Found.00 Found.01. etc. and one of them is usually close to the amount
of data I copied. (Just guessing here on the amount).
 
Old 04-30-2022, 12:10 PM   #13
boughtonp
Senior Member
 
Registered: Feb 2007
Location: UK
Distribution: Debian
Posts: 3,627

Rep: Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556Reputation: 2556

Is your aim to solve this issue, or do you only want to complain?

 
Old 04-30-2022, 12:20 PM   #14
michaelk
Moderator
 
Registered: Aug 2002
Posts: 25,783

Rep: Reputation: 5936Reputation: 5936Reputation: 5936Reputation: 5936Reputation: 5936Reputation: 5936Reputation: 5936Reputation: 5936Reputation: 5936Reputation: 5936Reputation: 5936
Obviously the latter. Switch to NTFS if you want compatibility or a linux filesystem if you do not. I've only needed exFAT support a few times and never had a problem.

If you need exFAT only use it with windows.
 
1 members found this post helpful.
Old 04-30-2022, 12:26 PM   #15
DrDwayne
Member
 
Registered: May 2010
Location: Wichita
Distribution: Ubuntu Debian
Posts: 155

Original Poster
Rep: Reputation: 4
Quote:
Originally Posted by boughtonp View Post
Is your aim to solve this issue, or do you only want to complain?

I would love to solve the problem. . .
I can't find answers on the internet.
I have quite a few larger USB drives and External Hard drives
that I am afraid to even plug in.

It is frustrating when you plug these things in, and lose your data
while trying to write to it.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Slackware 14.2 freezes when copying large exfat files linuxbird Slackware 4 09-08-2018 07:55 PM
dd to copy a thumb drive which I am live booted on to another thumb drive DrinkinHomeBrew Linux - Newbie 3 01-26-2015 04:52 PM
Formatting Flash drives / thumb drives lawrence_lee_lee Linux - Software 3 06-12-2008 02:28 AM
LXer: This week at LWN: Large pages, large blocks, and large problems LXer Syndicated Linux News 0 09-27-2007 11:40 AM
mknod and compress corrupting large files mhodson Linux - Software 0 03-27-2006 11:24 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 06:19 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration