CentOS7 fails to boot: "Failed to mount /sysroot"
This was a fully functioning server up until this afternoon. Got a strange error about something tieing up the processor while in a ssh session. Could not get any commands though. Had my GF shut it down (power off) as there was nothing I could do remotely.
When I turned it back on it looked like it was booting normally but then I received an error message "Failed to mount /sysroot" and I got dropped into an emergency terminal. There is a log file it created. Tried to cat it but way too big. just see the trailing end where it starts failing on /sysroot. Can't retrieve the log to show here (yet) cause I left my rescue USB at work (not my brightest moment). I will have the logs to post tomorrow evening I am going to work on it more in depth tomorrow, however if you guys could give me some ideas of what could be going on and where to look I would be thankful. I have never seen an error like that before. |
This could potentially be a problem with the RAM.
This sounds like it is using intird to boot the system which attempts to mount /sysroot to your RAM and then look for the fstab as well as other steps. I am not an expert, nor do I want to be, on systemd but it sounds like that is where the hangup is happening. If you cant get any other diagnostic info might be worth it to use just a single stick of known-good RAM and attempt to boot. And my personal rant: RHEL 6 FOR LIFE! BOYCOTT SYSTEMD! :D |
Quote:
I am getting the logs now. Might take me a bit. Haven't figured out how I am going to do this yet. Tempted to take the whole drive out and boot it in a VM to rule out any hardware issues. Anyway, rescue live being loaded onto a usb stick now... EDIT: I have ruled Hardware Fault in RAM. SystemRescueCD (or USB in this case) is running completely from the RAM with no issues EDIT: Having an issue mounting the BTRFS partition from inside SRCD. The exact error I am getting is: "couldn't mount because of unsupported optional features (40)" |
If your suspicious that the RAM is bad run Memtest overnight and see if it passes.
http://www.memtest.org/ |
1 Attachment(s)
Ram seems ok, but I do plan on running memtest tonight. Thank you for reminding me I don't have to download the whole UBCD for memtest lol.
I was finally able to grab that log file. I am posting it here as I am reading it. Have not gone over it all yet. EDIT: It almost seems like it is having issues mounting that BTRFS partition. Any utilities that can check that type of file-system for errors? (asking this pre-google-foo) EDIT: Post-google-foo: https://btrfs.wiki.kernel.org/index....ge/btrfs-check EDIT: Excerpt from the log: Code:
[ 2.544512] Apollo kernel: ------------[ cut here ]------------ |
Your Welcome:-
This was the only link remotely close to the Warning you have in the output of that log. http://www.spinics.net/lists/linux-btrfs/msg30844.html I'm not familiar with what BTRFS is or what the fault tolerance means, sorry. Maybe this WiKi will be of some use. https://btrfs.wiki.kernel.org/index.php/Main_Page Where you right in the middle of transferring a file from host to guest when you got that strange error about something tieing up the processor while in a ssh session? Here's information of a BTRFS Repair tool- http://www.phoronix.com/scan.php?pag...tem&px=MTA2MDI I have some experience with Boot Information Script. Its primary use is for troubleshooting booting problems. This might help us to figure out what's going on. When the script is done it will put a .txt file on your system. http://sourceforge.net/projects/bootinfoscript/ |
RAM Confirmed OK: 22 passes: 0 Errors
Working the BTRFS repair angle as I am unable to mount that partition by any means. Will also use the script. Will that script run fine from a rescue USB? There was some activity between the database and a program my co-worker is currently writing when this all started. I am allowing him limited access to the database while he learns Java. He has been working on it for well over a month without incident so I did not think it was a possible cause. Could some communication between his program and MariaDB cause this? |
Quote:
When I ran that script it placed the .txt file in my Downloads directory. I have never come across a program interfering with a data base but anything is possible. What program was your co worker using? How old is the HDD? I hope that btrfs check with the repair option works for you. Was the partition changed or modified in any way? Resized? |
I looked up the 2 warning in the output and I found this report.
Code:
2.544604] Apollo kernel: [<ffffffff8105dee1>] warn_slowpath_common+0x61/0x80 Sorry, I have never seen those warnings before. |
Quote:
The program is of his own design. He is learning Java and wanted to learn how to access and write to a database. Very basic. I really hope it works too. Have not had a chance to work on it today. I can only hope I get some time tomorrow. Partition was not modified. I leave that sort of thing alone |
Quote:
-:-When you have the time to go through the BTRFS repair let me know how it goes.-:- |
Quote:
Leaving it alone to see if it does anything else. I might just have to reinstall. And if that is the case I need to plan ahead better and figure out some redundancies. Never seen a FS just crap out like that and then can not be repaired. In fact, this is the first time I have heard of btrfs. EDIT: yep, so it did eventually finish but it didn't "repair" anything. Just had a long list of blocks that did not match what it expected. Will try with other options like scrub from here: https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs |
Starting a new post because I am going in a new direction.
Data does not appear to be recoverable. All that was lost was some hard work I put into the configuration so it was not a major deal. In fact, this just might be a blessing disguise. I originally installed and configured this server just to see what the new CentOS7 could do kept using it after as I really did like the OS (sorry Kustom42, looks like that puts me in the systemd camp). What I should have done was install it with specific options and configure it with specific goals in mind. Specifically, I should have told my friend to use a developmental environment for his java experimenting. I am not pointing fingers but I am fairly certain he was messing with that tool of his when this whole thing started. Also, I wanted to tighten down security so I can only access the server via ssh and ftp with a cert and key. What I am getting at is that this gives me the excuse to start over and plan my approach rather then going ad hoc and reacting to situations. This might be a single home server but I know I can do better. If I can not figure out how to recover the filesystem in by this time tomorrow I will close this thread as "Solved: FS Corruption" and move on to reinstallation |
Quote:
I lost 30 days worth of files once because I had to partake in a not so graceful shutdown. The system locked up on me and there was nothing I could do. The freeze damaged my files and I couldn't reboot- Unfortunately, I had to reinstall the OS. Any chance you have a CentOS Live CD? With it you can Rescue Installed System- http://isoredirect.centos.org/centos/6/isos/x86_64/ http://www.standalone-sysadmin.com/b...S-bootmenu.png I asked because I wasn't sure if the Live USB has that option or not. Sorry your going through this, I've been there and I know it's not fun at all- I downloaded the CentOS Bible 'PDF' if I find anything worthy of repairing your fs I'll post what I find. Hope the instructions on the Manpage of btrfs 'scrub' works for you. |
Quote:
-::-Blessings in disguise can be a good teacher and in some cases a lesson well learned.-::- Running a program in the improper environment most likely creates undesired effects. Tightening Security is always a plus when your running a enterprise. |
All times are GMT -5. The time now is 03:53 PM. |