How to remount root partition on system booted from external usb drive
 

Hello,
I have a setup with arch linux installed on external ssd as normal (non-live) installation.
I had to do that after my main laptop was broken. I don't want to put my drive into the new hardware, so I run it as external one for until the main will be repaired.

It boots as usual, everything is working fine until the usb drive reconnects for some reason (may be the physical connection problem). It probably mounts itself as read-only, the system freaks out and dies slowly from I/O errors process by process. I cannot issue commands from shell, even reboot. I have to physically reset the machine.

Is there a way to change its behaviour to auto remount as rw after reconnecting the USB and just keep running as usual?

I think the fstab is the file that i should modify, but cannot find a way how to do it properly. Please correct me if I'm wrong. My current fstab:

I don't think so.

I believe your problem is that the swapfile is also on root - and when that filesystem remounts RO (which normally CAN be repaired), the swapfile is now locked.... parts in memory and parts on disk...

And the activity outstanding on the swapfile can't be completed thus locking up the system.

You might try repartitioning and putting the swapfile as a self contained partition. I believe the swap activation would then complete (now being independent of the root filesystem) which may allow you to recover the system (a "mount -o remount,rw /" should work).

I would like to avoid mocking with this disk partition table for time being. As i have 4 gigs of ram on this temporary hardware and fairly lightweight system, I can disable the swap entirely and just keep my eye on memory usage while working. But after issuing "sudo swapoff -a" and effectively disabling all swap, the problem still occurs. After unplugging and plugging back again the system drive, it just dies. I even got a kernel panic for the first time on this system.

Your test was a good one.

It is possible that root cannot be disconnected once services have been started. Even without swap, a swap operation (actually a page read) is done to load binaries.

It is possible that the disk driver hasn't got sufficient error recovery implemented.

I believe an NFS root can disconnect, and reconnect successfully, which is why I mention "hasn't got sufficient error recovery". NFS has loads of such recovery, but I don't know about local disk drivers having that much.

There is some mount options that might change what is going on. From the mount manpage:
Since you mentioned getting a panic when you disabled swap, it is possible that the filesystem is directing the system to not continue. You might TRY (and note, this may cause other failures) to see what happens with "continue".

After testing it further, I've seen that random crashes happen a lot less frequently with swap off, but this may be just my luck. System still crashes after manual unplug though. Also system cannot wake from sleep, but there may be other causes (as slightly different hardware than usual).

As I want to be very cautious about my external drive, is it "safe enough" to set the errors=continue option? From what I understand, romount-ro is a safety valve in case something could go horribly wrong. And I still should be able to remount my partition from terminal into rw. From unknown reasons though, the disk cannot be even accessed, so it is probably not remounted at all, which effectively eliminates the possibility of issuing "mount" command. My wild guess is that not enough kernel/tools are prepared in RAM to recover from error. Is there a way to prepare the system for this situation? Just like some "live" distros can run without the usb stick after boot.

I've managed to get into text mode in the early stage of crash, and this is what i saw
Attachment 17384

There is... but it might take a good bit of work.

The key is the initrd file used at boot time. It is actually a very small root filesystem that gets copied into a RAMFS. Now once the real root filesystem is mounted, the initrd init then uses the "pivot" system call to exchange the initrd based root for the real root - then discards the initrd root and continues loading applications.

It should be possible to set up things such that the initrd root remains, and then uses the "real" root as an application mount (the way /usr was intended - an extension of the root). It will use up system memory (several MB, maybe as much 50? might even be more), but it would remain resident even if the USB disconnects, then reconnection events wouldn't need handling via page faults from the disk...

It would be a custom initrd, with more of the existing init. I know that root for most systems can be trimmed to about 3 GB, but that is too much memory to loose for many laptops.

I would expect the system to freeze, but hopefully recover since the udev handling would all be on the initrd (and hence in memory). Updating /etc might be tricky as you would have set it in the initrd used for booting (or have the initrd copy updated files from the real root). I don't know if a bind mount of /usr would be sufficient. A number of distributions have been putting things in /usr that really ought to be in the root directory.

Things like /tmp and /var might work (though the log files might cause a problem).

Slackware used to do this... They would boot the kernel from one floppy, and use a second floppy for root, and that second 1MB had sufficient tools to even install the system as well as recover one.

If you were willing to use a CD (or even DVD) as a root boot you should be able to put the everything else on the USB. I realize that is a bit of cribbing, but with a DVD you can have up to 3GB for root, with a CD up to 700 MB, and without using up your RAM.

That would be the log file problem.

Hopefully, it doesn't cause the log to get corrupted.