[SOLVED] How to recover data from corrupted NVMe disk

cacoo · 03-31-2020, 02:35 AM

I have Fedora 31 installed on my laptop (NVMe + LVM partitions) . It failed to boot today and I have tried several methods to recover on live CD but all failed. Is there anything I could do to recover my data?

Firstly I tried fsck, it failed quickly:

Code:

fsck -fy /dev/mapper/fedora-root
fsck from util-linux 2.34
e2fsck 1.45.3 (14-Jul-2019)
/dev/mapper/fedora-root: recovering journal
Superblock needs_recovery flag is clear, but journal has data.
Run journal anyway? yes

fsck.ext4: Input/output error while recovering journal of /dev/mapper/fedora-root
fsck.ext4: unable to set superblock flags on /dev/mapper/fedora-root


/dev/mapper/fedora-root: ********** WARNING: Filesystem still has errors **********

Then I tried to recover with backup superblock by e2fsck, it failed, too:

Code:

e2fsck -b 32768 /dev/mapper/fedora-root
e2fsck 1.45.3 (14-Jul-2019)
Superblock needs_recovery flag is clear, but journal has data.
Recovery flag not set in backup superblock, so running journal anyway.
/dev/mapper/fedora-root: recovering journal
e2fsck: Input/output error while recovering journal of /dev/mapper/fedora-root
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  +(819200--820231) +(884736--885767) +(1605632--1606663) +(4096000--4097031) +(7962624--7963655) +(11239424--11240455)
Fix<y>? yes to all
Free blocks count wrong for group #0 (23506, counted=23500).
Fix? yes
...
Padding at end of inode bitmap is not set. Fix? yes

Block bitmap differences: Group 0 block bitmap does not match checksum.
FIXED.
Error writing file system info: Input/output error

/dev/mapper/fedora-root: ***** FILE SYSTEM WAS MODIFIED *****

output from dmesg:

Code:

[ 3323.929740] blk_update_request: I/O error, dev nvme0n1, sector 2101248 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.929751] buffer_io_error: 190 callbacks suppressed
[ 3323.929754] Buffer I/O error on dev dm-2, logical block 0, lost async page write
[ 3323.934180] blk_update_request: I/O error, dev nvme0n1, sector 2101264 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.934192] Buffer I/O error on dev dm-2, logical block 2, lost async page write
[ 3323.938619] blk_update_request: I/O error, dev nvme0n1, sector 2101288 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.938637] Buffer I/O error on dev dm-2, logical block 5, lost async page write
[ 3323.943054] blk_update_request: I/O error, dev nvme0n1, sector 2109760 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.943066] Buffer I/O error on dev dm-2, logical block 1064, lost async page write
[ 3323.947484] blk_update_request: I/O error, dev nvme0n1, sector 6295808 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.947496] Buffer I/O error on dev dm-2, logical block 524320, lost async page write
[ 3323.951929] blk_update_request: I/O error, dev nvme0n1, sector 18878720 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.951942] Buffer I/O error on dev dm-2, logical block 2097184, lost async page write
[ 3323.956369] blk_update_request: I/O error, dev nvme0n1, sector 18878776 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.956386] Buffer I/O error on dev dm-2, logical block 2097191, lost async page write
[ 3323.960801] blk_update_request: I/O error, dev nvme0n1, sector 18878824 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.960819] Buffer I/O error on dev dm-2, logical block 2097197, lost async page write
[ 3323.965308] blk_update_request: I/O error, dev nvme0n1, sector 18879336 op 0x1:(WRITE) flags 0x800 phys_seg 3 prio class 0
[ 3323.965320] Buffer I/O error on dev dm-2, logical block 2097261, lost async page write
[ 3323.965332] Buffer I/O error on dev dm-2, logical block 2097262, lost async page write
[ 3323.969779] blk_update_request: I/O error, dev nvme0n1, sector 23073432 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0

Other information may help:

Code:

smartctl -a /dev/nvme0n1
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
- available spare has fallen below threshold
- media has been placed in read only mode

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x09
Temperature:                        0 Celsius
Available Spare:                    100%
Available Spare Threshold:          3%
Percentage Used:                    37%
Data Units Read:                    2,815,372 [1.44 TB]
Data Units Written:                 3,781,234 [1.93 TB]
Host Read Commands:                 160,034,805
Host Write Commands:                218,461,099
Controller Busy Time:               735
Power Cycles:                       558
Power On Hours:                     351
Unsafe Shutdowns:                   92
Media and Data Integrity Errors:    4
Error Information Log Entries:      37
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               0 Celsius
Temperature Sensor 2:               0 Celsius

Error Information (NVMe Log 0x01, max 4 entries)
Num   ErrCount  SQId   CmdId  Status  PELoc          LBA  NSID    VS
  0         37     0  0x002b  0xc004  0x000            0     0     -
  1         36     0  0x0023  0xc004  0x000            0     0     -
  2         35     0  0x0021  0xc005  0x000            0     0     -
  3         34     0  0x001e  0xc004  0x000            0     0     -

pan64 · 03-31-2020, 02:59 AM

I would try to dd it into another drive and use testdisk on the mirror. Otherwise it looks like this device is now died. (but accessible in read only mode, so you can make a backup).