LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 04-27-2020, 06:59 AM   #16
wpeckham
LQ Guru
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, VSIDO, tinycore, Q4OS,Manjaro
Posts: 5,627

Rep: Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695

Quote:
Originally Posted by Linux0110 View Post
Yes sure I did. I did this operation multiple times and the files get corrupted as soon as they are copied into the ext4 drive without even removing the pendrive. If I use FAT32 it all works fine. However not all files get corrupted (mp.3 and mp.4 istantly corrupted, Pictures in jpg seem to not get corrupted, Office documents are fine and finally PDF sometimes get corrupted, sometimes not.)
This seems to be ignored by others, so let me highlight it here.
If your data is instantly corrupted on copy, without the USB device being unmounted, physically removed, or in any way interrupted, then it is something about the USB device itself or the driver for it that is causing the corruption.
It could be the specific combination of driver and device, but that would be somewhat unusual.

I would ask about the kind of corruption, but that may not matter much at this point. I am interested in how you detect the corruption and how it presents to you. It is information you might want to add to this thread for consideration.

What are you using to make the copy? I would like to see the command line, if any. Perhaps something that verifies the copy would work better. (Rsync comes to mind.) Was this USB device prepared with any kind of encryption in mind?

See if a different device shows the same symptoms. If not, then try different file systems on THAT device.
As much as I would love to dig into the root cause, your purposes will be better served by finding a workaround.
 
1 members found this post helpful.
Old 04-27-2020, 08:23 AM   #17
rtmistler
Moderator
 
Registered: Mar 2011
Location: USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 9,882
Blog Entries: 13

Rep: Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930
Quote:
Originally Posted by wpeckham View Post
This seems to be ignored by others, so let me highlight it here.
I agree it's important, but I feel a few people have asked what they have done, and how they've detected these corruptions. Unfortunately there's no details about that.
 
Old 04-27-2020, 02:56 PM   #18
jefro
Moderator
 
Registered: Mar 2008
Posts: 21,982

Rep: Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626
I've had at least one brand of USB3 flash drive that simply would not work correctly on ext4. I assume it was the controller/caching as the next 20 brands worked correctly.

If the drive you have fails on ext4 then consider a new brand or use supported filesystem.
 
Old 04-27-2020, 03:52 PM   #19
crts
Senior Member
 
Registered: Jan 2010
Posts: 2,020

Rep: Reputation: 757Reputation: 757Reputation: 757Reputation: 757Reputation: 757Reputation: 757Reputation: 757
Quote:
Originally Posted by jefro View Post
I've had at least one brand of USB3 flash drive that simply would not work correctly on ext4.
Could you post the brand/model of the USB flash drive? I searched for usb flash drives that do not work with ext4 and it seems to be an actual issue. I found this thread that describes a similiar issue as OP:
https://superuser.com/questions/1317...format-as-ext4

This one seems to be a driver issue and in both cases the problem is seems to be limited to ext4 format.

@OP please post brand/model of your USB and the Distro you are using.
 
Old 04-27-2020, 08:00 PM   #20
jefro
Moderator
 
Registered: Mar 2008
Posts: 21,982

Rep: Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626
I'd have to look at past newegg purchases.

I am sure some posts at LQ have had this same issue that was resolved by changing the drive brand.

There was never any tests that I did to try to resolve the problem. Too much trouble but I am pretty sure it was limited to one big name brand. It could be that the motherboard or any part of the system could have caused it. All I know is I could not use it as a bootable linux distro on ext4. xfs worked and I think I tried btrfs that failed but it was a while back.
 
Old 04-27-2020, 08:41 PM   #21
Linux0110
Member
 
Registered: Mar 2020
Posts: 69

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by wpeckham View Post
This seems to be ignored by others, so let me highlight it here.
If your data is instantly corrupted on copy, without the USB device being unmounted, physically removed, or in any way interrupted, then it is something about the USB device itself or the driver for it that is causing the corruption.
It could be the specific combination of driver and device, but that would be somewhat unusual.

I would ask about the kind of corruption, but that may not matter much at this point. I am interested in how you detect the corruption and how it presents to you. It is information you might want to add to this thread for consideration.

What are you using to make the copy? I would like to see the command line, if any. Perhaps something that verifies the copy would work better. (Rsync comes to mind.) Was this USB device prepared with any kind of encryption in mind?

See if a different device shows the same symptoms. If not, then try different file systems on THAT device.
As much as I would love to dig into the root cause, your purposes will be better served by finding a workaround.

Thank you to have highlighted the points. When I try to open the corrupted files it tells me that it is an unrecognized file format or it tells straight that the file is corrupted. The unrecognized file format thing is bu#$%^^&t because if I download the same file with the same format and open it in the hard drive it works fine. It is not a problem of a USB in particular it happens with all my pendrives if formatted to ext4. I also did backups with linux mint backup tool and when I remove the pendrive (unmounting it and oblviously formatted to ext4) and then instert it the files are just gone or maybe there are some left for unknown reasons. As I said before if I use FAT32 on the same devices it all works perfectly without any kind of issues or data loss.
 
Old 04-27-2020, 08:43 PM   #22
Linux0110
Member
 
Registered: Mar 2020
Posts: 69

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by jefro View Post
I've had at least one brand of USB3 flash drive that simply would not work correctly on ext4. I assume it was the controller/caching as the next 20 brands worked correctly.

If the drive you have fails on ext4 then consider a new brand or use supported filesystem.
It doesn't fail in just one drive it fails on all my pendrives that are made by different companies they are not all the same.
 
Old 04-27-2020, 08:45 PM   #23
Linux0110
Member
 
Registered: Mar 2020
Posts: 69

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by jefro View Post
I'd have to look at past newegg purchases.

I am sure some posts at LQ have had this same issue that was resolved by changing the drive brand.

There was never any tests that I did to try to resolve the problem. Too much trouble but I am pretty sure it was limited to one big name brand. It could be that the motherboard or any part of the system could have caused it. All I know is I could not use it as a bootable linux distro on ext4. xfs worked and I think I tried btrfs that failed but it was a while back.
The pendrives are not of the same brand they are made by different companies.
 
Old 04-27-2020, 08:52 PM   #24
rtmistler
Moderator
 
Registered: Mar 2011
Location: USA
Distribution: MINT Debian, Angstrom, SUSE, Ubuntu, Debian
Posts: 9,882
Blog Entries: 13

Rep: Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930Reputation: 4930
All I got is I've never run into it and used many a USB thumbstick exactly as I say.

Has anyone ever torn some apart? We did a few and amazingly they constituted of a micro SD card coupled with connections to attach it to the USB connector.

Sort of amazed us.

Kingston, Sansa, Memorex. Those were some brands I used. Whatever our parts buyer got when he ordered a bunch. We made some prototype products with USB sticks included to store the data, provide a mountable file system outside the product, and be available to be removed if the manufacturer so was inclined. Couple hundred prototypes, never had data problems, with the exception that I had to work a trick to ensure files would be available for offloading right after a field activity was completed.

TMI:
To fix I stored all data into a temp file, upon activity completion I forked a child to rename the temp to the final record, then exited the child. Upon successful termination signal from the child all file handles had to be resolved for that child, that made the file available and not corrupt.
 
Old 04-27-2020, 09:09 PM   #25
OliverYY
LQ Newbie
 
Registered: Jan 2020
Posts: 15

Rep: Reputation: Disabled
Unless the drive itself is faulty it's most likely the fault of your choice of file systems. You should perform a chkdisk/scandisk from within windows to check for and correct errors.

The drive has likely been corrupted either by trying to copy a file over 4 GB in size, or by filenames that contain characters that are not permitted on a windows file system.
 
Old 04-27-2020, 09:12 PM   #26
jefro
Moderator
 
Registered: Mar 2008
Posts: 21,982

Rep: Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626
Rather doubt it is ext4 filesystem currently.

I can only say that one name brand did I ever have any issue. If multiple name brand usb flash drives fail on ext4 then I'd have to assume some other issue. Ext4 has been used by too many people on usb flash to claim it is a problem. (I'd think)

All filesystems get some updates every so often. Use latest version on modern distro if possible.

I guess rtmistler suggestion of tear down or other test could reveal same chips causing this issue.

Last edited by jefro; 04-27-2020 at 09:14 PM.
 
Old 04-28-2020, 06:45 AM   #27
wpeckham
LQ Guru
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, VSIDO, tinycore, Q4OS,Manjaro
Posts: 5,627

Rep: Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695
OK, bottom line, if you have a device where EXT4 does not work, stop using EXT4 on that device and use what WORKS. We have not nailed down a cause, but it is likely to be the driver or software on your machine. (Else it would not affect all USB flash devices.)

FAT systems are to be avoided on all semi permanent storage due to the shortcomings of that format. Flash is not semi permanent storage, it is strictly temporary and untrustworthy. For flash use fat is just fine. Use the EXFAT to get expanded file and system sizes appropriate for modern capacity flash. You may want to run tests to see if other file systems suitable for USB FLASH use perform well and are suitable for your use on your devices on that system. I would be interested in running such tests myself, but I cannot replicate your issue.

Because flash is more prone to failure than other media, if you use it for backups make sure that you have multiple copies of your backups so that WHEN (not if) one fails you do not lose that data. This is good practice for ANY backup plan no matter what the backup media, but a bit more critical for flash media.
 
Old 04-28-2020, 10:50 AM   #28
DavidMcCann
LQ Veteran
 
Registered: Jul 2006
Location: London
Distribution: PCLinuxOS, Debian
Posts: 6,142

Rep: Reputation: 2314Reputation: 2314Reputation: 2314Reputation: 2314Reputation: 2314Reputation: 2314Reputation: 2314Reputation: 2314Reputation: 2314Reputation: 2314Reputation: 2314
Since I have no use for Windows, I've always re-formated by ext2. I chose that because the fact that it doesn't maintain a journal means that it imposes less wear on the drive; I also mount with the parameters noatime,nodiratime.

USB flash drives will eventually fail. My rule when there's a problem is firstly to use fsck; if it happens again, reformat; third time, chuck it.

There are some fakes sold on the internet, particularly of high-capacity ones. You can check them with the utility f3:
https://www.linuxbabe.com/command-li...usb-test-linux
 
Old 04-28-2020, 11:33 AM   #29
sp331yi
Member
 
Registered: Mar 2020
Location: NM
Distribution: antiX 19.2 | Slacko pup | Miyo
Posts: 104

Rep: Reputation: Disabled
Hell, I like ext2 -- tell me what's wrong with that, especially for encryption, please.
 
Old 04-28-2020, 03:39 PM   #30
jefro
Moderator
 
Registered: Mar 2008
Posts: 21,982

Rep: Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626Reputation: 3626
sp331yi, while I would only use ext2 on some very old distro's that doesn't mean your use is bad. You are the one who decides.

Let me ask this of the audience. Has anyone ever had the same problem with ext4 on multiple usb flash drives ... ever?

Last edited by jefro; 04-28-2020 at 07:56 PM.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Cloning bootable USB flash drive onto larger flash drive - best method? cilbuper Linux - General 5 02-26-2014 09:51 AM
Can't format USB flash drive, write protection krisbitner Linux - Hardware 5 02-07-2007 03:18 PM
How to format a usb flash drive eypros Linux - Software 4 04-19-2006 01:06 PM
FLASH FLASH XP Home and Samba Fix FLASH FLASH zollodav Linux - Networking 1 02-19-2006 01:16 AM
How to format a USB flash drive? Erik_the_Red Linux - Hardware 4 08-29-2005 12:47 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 10:25 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration