LinuxQuestions.org
Help answer threads with 0 replies.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Fedora
User Name
Password
Fedora This forum is for the discussion of the Fedora Project.

Notices


Reply
  Search this Thread
Old 04-29-2018, 09:26 AM   #1
przemhb
LQ Newbie
 
Registered: Apr 2018
Posts: 15

Rep: Reputation: Disabled
FC27 Failed to mount POSIX Message Queue, Kernel File System... after hard reset - permission denied


My Fedora 27 x64 (4.14.14-300) runs on GB-BACE-3160 Brix PC. The PC uses 320GB HDD. The system is used as a simple server, so it works 24h.

From time to time - a few weeks or so - it freezes. HDD keeps writing something (repetitive, although irregular sound). No reaction to keyboard, mouse, remote SSH access. I have tried to leave it overnight, but it does not recover, so I am forced to hard reset it.

This time something has broken. Booting fails. It says:
Failed to mount POSIX Message Queue File System,
Failed to start Remount and Kernel File Systems,
Failed to mount Kernel Debug File System,
Failed to mount Huge Pages File System

and lots of other failures comes after these.
See https://photos.app.goo.gl/qBUxT40zA2MTLTwO2

In emergency mode I have ran fsck. It says file systems are clean. SMART tests (Smartctl -h, short and long) found the disk to be perfectly healthy.

In all these cases
Failed at step EXEC spawning /usr/bin/mount: Permission denied
is given as a reason.

What happen and how to fix this? What is causing system to freeze from time to time? How to diagnose it and prevent it?

I will be thankful for your help.
 
Old 04-30-2018, 06:53 PM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,126

Rep: Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120
If it fails regularly like that I'd suspect temperature - or maybe (less likely) memory. Should be something in the journal.
Boot an install iso and see if you can chroot into the on-disk F27 - that at least will enable you to see if the system/data looks ok. Maybe the initrd got corrupted by the automatic fsck at boot. Use dracut from the chroot to rebuild the initramfs and retry a boot.
Might be worth setting up a script to save the temps every hour or so - include the time.
 
Old 05-01-2018, 03:39 AM   #3
przemhb
LQ Newbie
 
Registered: Apr 2018
Posts: 15

Original Poster
Rep: Reputation: Disabled
First of all thank you very much for your reply.

I have used F27 in Live mode booted from USB stick to take a look at those not mounted partitions. They appeared as ready to mount disks on XFCE desktop. I was able to mount all of them and they looked just fine.
I am not familiar with chroot. I have never used it.
Quote:
Use dracut from the chroot to rebuild the initramfs and retry a boot.
I'm not sure how to do this. Could you be so kind and explain it or point to some references?
 
Old 05-01-2018, 04:47 AM   #4
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,126

Rep: Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120
Just looking to see if the files are listed is not enough - you need to jump from the liveCD into the on-disk system. This is what a chroot does - you actually run from the disk so you can run file managers and pdf readers or whatever to ensure they run ok. Have a read of this for some background - you only need to do the bit under "Performing the chroot", but don't need the network, so maybe skip resolv.conf. Once in the chroot, run some programs, and when happy, simply run "sudo dracut" (no quotes) from a terminal. The Fedora wiki is a mess at the moment - try "man dracut" for info - use "q" to quit.

Note my sigline - you really need to ensure you have a valid backup prior to messing any more.
 
Old 05-01-2018, 12:09 PM   #5
przemhb
LQ Newbie
 
Registered: Apr 2018
Posts: 15

Original Poster
Rep: Reputation: Disabled
I have:
1. Backup boot, root, etc and other important files.
2. Followed steps described in "Performing chroot".
3. Checked mounted files. Didn't noticed anything wrong.
4. Ran dracut. It said "Cannot find module directory /lib/modules/4.13.9-300.fc27.x86_64/", so apparently it wanted to use Live kernel instead of the newest 4.15.14-300.fc27.x86_64 from mounted /mnt/boot
5. Checked kernel: uname -r -> 4.13.9-300.fc27.x86_64
6. cd into /mnt/boot
7. Ran dracut initramfs-4.15.14-300.fc27.x86_64.img 4.15.14-300.fc27.x86_64 --force
8. Verified initramfs was updated
9. exit chroot, umounted boot, proc, sys, dev, /mnt and rebooted.
Unfortunately it didn't change anything. The problem persists.

I have updated the system a few times, so I have 3 kernels. I have tried to boot each of them, but the system behaves just the same. I've added a movie showing system boot to the shared album linked in my initial post.

Have I done something wrong or simply it isn't the case? What else could I try?

Code:
[root@localhost /]# fdisk -l
Disk /dev/sda: 298.1 GiB, 320072933376 bytes, 625142448 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7fc39067

Device     Boot   Start       End   Sectors   Size Id Type
/dev/sda1  *       2048   2099199   2097152     1G 83 Linux
/dev/sda2       2099200 625141759 623042560 297.1G 8e Linux LVM


Disk /dev/sdb: 7.2 GiB, 7756087296 bytes, 15148608 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x193f291a

Device     Boot Start      End  Sectors  Size Id Type
/dev/sdb1  *       63 15148223 15148161  7.2G  b W95 FAT32


Disk /dev/mapper/fedora-swap: 2.1 GiB, 2214592512 bytes, 4325376 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/fedora-home: 96 GiB, 103062437888 bytes, 201293824 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/fedora-root: 50 GiB, 53687091200 bytes, 104857600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/fedora-fedora--home--extra: 149 GiB, 160029474816 bytes, 312557568 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mmcblk1: 117.2 GiB, 125829120000 bytes, 245760000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x749b03a3

Device         Boot Start       End   Sectors   Size Id Type
/dev/mmcblk1p1 *     2048 245746304 245744257 117.2G  7 HPFS/NTFS/exFAT


Disk /dev/loop0: 1.1 GiB, 1191088128 bytes, 2326344 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop1: 5 GiB, 5370806272 bytes, 10489856 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop2: 990 MiB, 1038090240 bytes, 2027520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop3: 32 GiB, 34359738368 bytes, 67108864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/live-rw: 5 GiB, 5370806272 bytes, 10489856 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/live-base: 5 GiB, 5370806272 bytes, 10489856 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdc: 55.9 GiB, 60011642880 bytes, 117210240 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000001

Device     Boot Start       End   Sectors  Size Id Type
/dev/sdc1          63 117210239 117210177 55.9G  7 HPFS/NTFS/exFAT

========

[root@localhost /]# mount -t ext4 /dev/mapper/fedora-root /mnt
[root@localhost /]# mount -t proc proc /mnt/proc
[root@localhost /]# mount -t sysfs sys /mnt/sys
[root@localhost /]# mount -o bind /dev /mnt/dev
[root@localhost /]# mount -t ext4 /dev/sda1 /mnt/boot
[root@localhost /]# chroot /mnt /bin/bash

========

[root@localhost /]# grep -v rootfs /proc/mounts
/dev/mapper/fedora-root / ext4 rw,seclabel,relatime,data=ordered 0 0
proc /proc proc rw,relatime 0 0
sys /sys sysfs rw,seclabel,relatime 0 0
devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=1918656k,nr_inodes=479664,mode=755 0 0
/dev/sda1 /boot ext4 rw,seclabel,relatime,data=ordered 0 0

[root@localhost /]# cat /etc/mtab
/dev/mapper/fedora-root / ext4 rw,seclabel,relatime,data=ordered 0 0
proc /proc proc rw,relatime 0 0
sys /sys sysfs rw,seclabel,relatime 0 0
devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=1918656k,nr_inodes=479664,mode=755 0 0
/dev/sda1 /boot ext4 rw,seclabel,relatime,data=ordered 0 0

========

[root@localhost /]# dracut
dracut: No '/dev/log' or 'logger' included for syslog logging
dracut: Cannot find module directory /lib/modules/4.13.9-300.fc27.x86_64/
dracut: and --no-kernel was not specified

========

[root@localhost /]# ls /boot
05a59e1dba434a949729623eed74773e
1718824b58024a4da9596917fb7e244f
config-4.14.16-300.fc27.x86_64
config-4.15.13-300.fc27.x86_64
config-4.15.14-300.fc27.x86_64
efi
elf-memtest86+-5.01
extlinux
grub2
initramfs-0-rescue-05a59e1dba434a949729623eed74773e.img
initramfs-4.14.16-300.fc27.x86_64.img
initramfs-4.15.13-300.fc27.x86_64.img
initramfs-4.15.14-300.fc27.x86_64.img
lost+found
memtest86+-5.01
System.map-4.14.16-300.fc27.x86_64
System.map-4.15.13-300.fc27.x86_64
System.map-4.15.14-300.fc27.x86_64
vmlinuz-0-rescue-05a59e1dba434a949729623eed74773e
vmlinuz-4.14.16-300.fc27.x86_64
vmlinuz-4.15.13-300.fc27.x86_64
vmlinuz-4.15.14-300.fc27.x86_64

========

[root@localhost /]# uname -r
4.13.9-300.fc27.x86_64

[root@localhost /]# dracut initramfs-4.15.14-300.fc27.x86_64.img  4.15.14-300.fc27.x86_64 --force

========

[root@localhost liveuser]# umount /mnt/boot
[root@localhost liveuser]# umount /mnt/{proc,sys,dev}
[root@localhost liveuser]# umount /mnt
[root@localhost liveuser]# cd /mnt
[root@localhost mnt]# reboot
 
Old 05-01-2018, 12:33 PM   #6
przemhb
LQ Newbie
 
Registered: Apr 2018
Posts: 15

Original Poster
Rep: Reputation: Disabled
Could it be all those file systems cannot be mounted as system doesn't recognize them as it's own? Maybe it sees them as foreign partitions and that is why permission is denied to mount them?
 
Old 05-07-2018, 02:24 PM   #7
przemhb
LQ Newbie
 
Registered: Apr 2018
Posts: 15

Original Poster
Rep: Reputation: Disabled
I've changed SELinux to Permissive mode in /etc/selinux/config and system booted, so I've checked local modifications to SELinux config files and file_contexts:
Code:
#semanage fcontext -C -l
fcontext SELinux   type        Context
/usr/bin/mount     all files   system_u:object_r:samba_share_t:s0
I have noticed context for the mount is suspicious.

The problem was, indeed, caused by improper file context of the /usr/bin/mount file: samba_share_t.

The file context change wasn't caused by some error due to hard reset, but... by my imprudent decision to follow the first suggestion of SELinux Alert Browser.
This first suggestion was to change /usr/bin/mount file context to samba_share_t to allow smbd to access getattr.

The solution was:
  1. to delete invalid file context, restore default and relabel the file:
    Code:
    [root@atlas ~]# ls -Z /usr/bin/mount
    system_u:object_r:samba_share_t:s0 /usr/bin/mount
    [root@atlas ~]# semanage fcontext -d /usr/bin/mount
    [root@atlas ~]# restorecon -v /usr/bin/mount
    Relabeled /usr/bin/mount from system_u:object_r:samba_share_t:s0 to system_u:object_r:mount_exec_t:s0
    [root@atlas ~]# ls -Z /usr/bin/mount
    system_u:object_r:mount_exec_t:s0 /usr/bin/mount
  2. switch SELinux into enforcing mode
  3. reboot system.
 
1 members found this post helpful.
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Slow posix message queue? slash8 Programming 1 09-14-2013 07:23 AM
POSIX Message Queue LinuxDreams Linux - Newbie 2 08-20-2013 05:35 AM
mount : ... failed, reason given by server: Permission denied knaries2000 Linux - Networking 4 06-25-2013 11:33 AM
Permission Denied Error Message when trying to mount an NFS share kaplan71 Fedora 1 02-14-2009 07:26 AM
how to clear a posix message queue navalbabu Programming 5 01-11-2008 01:37 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Fedora

All times are GMT -5. The time now is 04:55 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration