Red Hat 6.5

paulfoel · 03-10-2015, 07:33 AM

Apart from boot.log (which has no times in) and /var/log/messages, what can I look at to determine why this server is taking so long to reboot?

paulfoel · 03-10-2015, 07:36 AM

Code:

Mar 10 06:30:35 server1 kernel: Kernel logging (proc) stopped.
Mar 10 06:30:35 sever1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="1317" x-info="http://www.rsyslog.com"] exit
ing on signal 15.
Mar 10 07:25:04 server1 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Mar 10 07:25:04 server1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="1317" x-info="http://www.rsyslog.com"] start

Extract from /var/log/messages. As you can see, there are 55 mins where nothing appears to happen. Any way to increase log level for this?

linosaurusroot · 03-10-2015, 07:40 AM

It's not so much log level but you're waiting for the new logging to start.

paulfoel · 03-10-2015, 08:41 AM

Quote:

Originally Posted by linosaurusroot

It's not so much log level but you're waiting for the new logging to start.

So how can I determine whats going on for these 55 minutes?

linosaurusroot · 03-10-2015, 09:29 AM

Quote:

Originally Posted by paulfoel

So how can I determine whats going on for these 55 minutes?

Are you present with the server and screen and keyboard?

paulfoel · 03-10-2015, 11:14 AM

Quote:

Originally Posted by linosaurusroot

Are you present with the server and screen and keyboard?

I have remote access. Its a live server so I can't reboot just now.

Its scheduled to reboot once a week at 6-30am.

linosaurusroot · 03-10-2015, 12:04 PM

Do you have a similar system for practice?
Has this slow boot problem only appeared recently?

John VV · 03-10-2015, 03:02 PM

is "bootchart" installed ?
http://www.bootchart.org/

use that to get a log and a SVG image you can look at

in more newer OS's this is built in ( RHEL7)
systemd-analyze

paulfoel · 03-11-2015, 04:37 AM

Quote:

Originally Posted by linosaurusroot

Do you have a similar system for practice?
Has this slow boot problem only appeared recently?

No test system. Not sure how long its been like this - probably a long time. I've only had visibility of this server for the past few weeks.

paulfoel · 03-11-2015, 04:44 AM

Quote:

Originally Posted by John VV

is "bootchart" installed ?
http://www.bootchart.org/

use that to get a log and a SVG image you can look at

in more newer OS's this is built in ( RHEL7)
systemd-analyze

Thanks. Had a look at this - looks good.

BUT, I wont get authorisation to install this on a live server unfortunately. Is there any way to check whats going on without installing additional tools?

linosaurusroot · 03-11-2015, 10:10 AM

Quote:

Originally Posted by paulfoel

No test system. Not sure how long its been like this - probably a long time. I've only had visibility of this server for the past few weeks.

Convince the boss you need a lifelike test system, make one from new and see if it has this boot problem. If it doesn't swap them.

frankbell · 03-11-2015, 09:34 PM

What are specs of this system and what is the usage environment?

One of the members of my LUG is an accomplished Linux sysadmin who has recently acquired (within the past year) some very brawny servers that run RHEL (I forget the manufacturer). It's been a while since he demonstrated them at a LUG meeting, but they have 64 GB RAM and serve about 150 users in a thin-client environment.

They take 20 minutes to boot to login from power on. That is their normal.

linosaurusroot · 03-12-2015, 05:32 AM

Quote:

Originally Posted by frankbell

They take 20 minutes to boot to login from power on. That is their normal.

That's a good point. I've seen several minutes (~10?) in bios-type activity before starting RH. If this is confirmed by OP it may be unavoidable until the next hardware change.

GaWdLy · 03-12-2015, 09:03 PM

Somebody needs to sit in front of it to see what takes it so long. It's pretty likely that there's an automated fsck going on (look in /etc/fstab to see). Larger servers take up to ~20 minutes to test RAM, engage and test RAIDs, but many servers need to fsck one filesystem or another at boot-time, and that could take quite a bit of time.

You really won't know until someone hooks a console up to it and reports back to you.

speirs1 · 03-13-2015, 06:10 PM

You said you only have ssh access? Does the server have a BMC? iLo, Drac etc. Redirect serial out to the Virtual Serial Port linking the BMC and server. You can then ssh to the BMC and view everything as if you were sitting at a console. Also remove "rhgb quiet" from the kernel boot parameter if you haven't already. /etc/grub/grub.conf

Red Hat has some nice articles on the VSP thing. Step by step even for the Dell/HP side.