LinuxQuestions.org
Visit Jeremy's Blog.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Ubuntu
Reload this Page speedup boot process
User Name
Password
Ubuntu This forum is for the discussion of Ubuntu Linux.

Notices


Reply
  Search this Thread
Old 07-29-2012, 09:48 AM   #1
mahmoodn
Member
 
Registered: May 2010
Posts: 427

Rep: Reputation: 16
speedup boot process

This chart shows the boot process of our server. Couple of questions:

http://postimage.org/image/kr9omiuqf/

1) Can you state what at the top 10 most time consuming services? I can not figure out.

2) Where is the network service? I want to know when it is started?

3) Can you explain why the ssh service start with such a long delay?
 
Old 07-29-2012, 11:34 AM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by mahmoodn View Post
what at the top 10 most time consuming services?
Any service that starts at roughly 3s until the end of the chart and is not a kernel thread?

Then time is just one aspect. For instance note a huge I/O wait spike at 9s when the system runs fsck? This should not happen every boot but running it does affect other processes. Also note I/O wait and CPU utilization increase at 40s when NFS starts to Do Stuff. Besides the chart will never be complete as similar parallel running processes are collapsed and idle background and short-lived processes removed.


Quote:
Originally Posted by mahmoodn View Post
Where is the network service? I want to know when it is started?
(Something driving) "ifup" at 11.5?


Quote:
Originally Posted by mahmoodn View Post
Can you explain why the ssh service start with such a long delay?
No. Unless the startup script gives you clues you'll need ssh daemon log, 'ssh -d -d -d' or strace output.
 
Old 07-29-2012, 11:55 AM   #3
mahmoodn
Member
 
Registered: May 2010
Posts: 427

Original Poster
Rep: Reputation: 16
Quote:
For instance note a huge I/O wait spike at 9s when the system runs fsck?
There is a fsck and fsck.ext3 activity around 9s
Is that really a huge gap?

Quote:
Also note I/O wait and CPU utilization increase at 40s when NFS starts to Do Stuff.
Sorry it seems that we read the chart differently. at 40s, there is no activity.

Quote:
No. Unless the startup script gives you clues you'll need ssh daemon log, 'ssh -d -d -d' or strace output.
As you can see at 57s the sshd is started. The question is which process causes the delay for sshd? in another word, the sshd service is dependent on which service(s)?
 
Old 07-29-2012, 12:54 PM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by mahmoodn View Post
Is that really a huge gap?
No it is not "a big gap". But when a process hogs resources other processes experience starvation, albeit for a short period of time.


Quote:
Originally Posted by mahmoodn View Post
at 40s, there is no activity.
So what is your explanation for the colored thingies in the bottom nfsd process bar then?


Quote:
Originally Posted by mahmoodn View Post
As you can see at 57s the sshd is started.
No, the service starts at 13 something seconds.


Quote:
Originally Posted by mahmoodn View Post
The question is which process causes the delay for sshd? in another word, the sshd service is dependent on which service(s)?
You have not shown any diagnostics to be able to conclude any other process is the cause of the delay. Anyway, check the "start on" line in /etc/init/ssh.conf to find out after which services the sshd service is started.
 
Old 07-29-2012, 01:02 PM   #5
mahmoodn
Member
 
Registered: May 2010
Posts: 427

Original Poster
Rep: Reputation: 16
Quote:
So what is your explanation for the colored thingies in the bottom nfsd process bar then?
You are right. I didn't notice that


Quote:
No, the service starts at 13 something seconds.
So why there is a really a huge gap for sshd from 13s to 57s ?

the ssd.conf file is:
Code:
description     "OpenSSH server"

start on filesystem or runlevel [2345]
stop on runlevel [!2345]

respawn
respawn limit 10 5
umask 022

# 'sshd -D' leaks stderr and confuses things in conjunction with 'console log'
console none

pre-start script
    test -x /usr/sbin/sshd || { stop; exit 0; }
    test -e /etc/ssh/sshd_not_to_be_run && { stop; exit 0; }
    test -c /dev/null || { stop; exit 0; }

    mkdir -p -m0755 /var/run/sshd
end script

# if you used to set SSHD_OPTS in /etc/default/ssh, you can change the
# 'exec' line here instead
exec /usr/sbin/sshd -D
There are "pre-start scripts" but I can not figure out them
 
Old 07-29-2012, 01:20 PM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by mahmoodn View Post
So why there is a really a huge gap for sshd from 13s to 57s ?
See the last line of my first post. Unless syslog shows clues set SSHD_OPTS in /etc/default/ssh for debug mode and change the "exec" line switches in /etc/init/ssh.conf to the same, reboot and use bootchartd again.
 
Old 07-29-2012, 01:25 PM   #7
mahmoodn
Member
 
Registered: May 2010
Posts: 427

Original Poster
Rep: Reputation: 16
the last line is executed always with -D
So what should I really do?
only set SSHD_OPTS in /etc/default/ssh and reboot?
 
Old 07-29-2012, 02:11 PM   #8
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
I wrote "set SSHD_OPTS in /etc/default/ssh for debug mode and change the "exec" line switches in /etc/init/ssh.conf to the same".
 
Old 07-29-2012, 07:54 PM   #9
mahmoodn
Member
 
Registered: May 2010
Posts: 427

Original Poster
Rep: Reputation: 16
Quote:
set SSHD_OPTS in /etc/default/ssh for debug mode
done by changing
SSHD_OPTS=
to
SSHD_OPTS=1

Quote:
change the "exec" line switches in /etc/init/ssh.conf to the same
This is what I really con not understand!!
exec /usr/sbin/sshd -D
Change to what?? Isn't -D the debug mode?

Do you mean
exec /usr/sbin/sshd $SSHD_OPTS
??
Last edited by mahmoodn; 07-29-2012 at 07:56 PM.
 
Old 07-29-2012, 08:09 PM   #10
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by mahmoodn View Post
done by changing
SSHD_OPTS=
to
SSHD_OPTS=1
I suggested you use "-d -d -d" as in SSHD_OPTS="-d -d -d"?


Quote:
Originally Posted by mahmoodn View Post
This is what I really con not understand!!
exec /usr/sbin/sshd -D
Change to what??
Try "exec /usr/sbin/sshd -D -d -d -d"?


Quote:
Originally Posted by mahmoodn View Post
Isn't -D the debug mode?
See 'man sshd'?


Quote:
Originally Posted by mahmoodn View Post
Do you mean
exec /usr/sbin/sshd $SSHD_OPTS
That should work if you source /etc/default/ssh on the line before execing sshd.
 
Old 07-30-2012, 09:07 AM   #11
mahmoodn
Member
 
Registered: May 2010
Posts: 427

Original Poster
Rep: Reputation: 16
So I changed and rebooted.
Where are the debug information stored?
When I login, I see
Code:
debug3: Copy environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
debug3: Copy environment: LANG=en_US.UTF-8
debug3: Copy environment: XDG_SESSION_COOKIE=de692068037f222a35a95f874dc6aed9-1343657233.852497-1575243254
Environment:
  USER=mahmood
  LOGNAME=mahmood
  HOME=/home/mahmood
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
  MAIL=/var/mail/mahmood
  SHELL=/bin/bash
  SSH_CLIENT=XXX.2.25.130 34906 22
  SSH_CONNECTION=XXX.2.25.130 34906 YYY.225.39.105 22
  SSH_TTY=/dev/pts/0
  TERM=xterm
  DISPLAY=localhost:10.0
  LANG=en_US.UTF-8
  XDG_SESSION_COOKIE=de692068037f222a35a95f874dc6aed9-1343657233.852497-1575243254
Running /usr/bin/xauth remove unix:10.0
/usr/bin/xauth add unix:10.0 MIT-MAGIC-COOKIE-1 1c139183ebfd70fb06ff7710b0d22418
Last edited by mahmoodn; 07-30-2012 at 09:10 AM.
 
Old 07-30-2012, 09:19 AM   #12
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
sshd sends verbose debug output to the system log.
 
Old 07-30-2012, 12:18 PM   #13
mahmoodn
Member
 
Registered: May 2010
Posts: 427

Original Poster
Rep: Reputation: 16
I don't know what mistake i made that I am not able to ssh remotely. I have to go to server room in order to revert the changes...!!!
Last edited by mahmoodn; 07-30-2012 at 12:25 PM.
 
Old 07-31-2012, 01:18 PM   #14
mahmoodn
Member
 
Registered: May 2010
Posts: 427

Original Poster
Rep: Reputation: 16
As I modify the scripts (adding -d -d -d), I can no longer ssh to the server. Hopefully there is a web interface...
In the syslog, I see:

Code:
cat /var/log/syslog | grep ssh
Jul 31 22:45:47 hpclab kernel: [  221.396508] init: ssh main process ended, respawning
Jul 31 22:45:52 hpclab kernel: [  227.030543] init: ssh main process (2857) terminated with status 255
Jul 31 22:45:52 hpclab kernel: [  227.030620] init: ssh main process ended, respawning
Jul 31 22:45:57 hpclab kernel: [  231.846283] init: ssh main process (2862) terminated with status 255
Jul 31 22:45:57 hpclab kernel: [  231.846357] init: ssh main process ended, respawning
Jul 31 22:46:04 hpclab kernel: [  238.296699] init: ssh main process (2866) terminated with status 255
Jul 31 22:46:04 hpclab kernel: [  238.296772] init: ssh main process ended, respawning
Jul 31 22:46:07 hpclab kernel: [  242.059518] init: ssh main process (2881) terminated with status 255
Jul 31 22:46:07 hpclab kernel: [  242.059592] init: ssh main process ended, respawning
Jul 31 22:46:11 hpclab kernel: [  245.377472] init: ssh main process (2903) terminated with status 255
Jul 31 22:46:11 hpclab kernel: [  245.377556] init: ssh main process ended, respawning
Jul 31 22:46:15 hpclab kernel: [  250.058691] init: ssh main process (2983) terminated with status 255
Jul 31 22:46:15 hpclab kernel: [  250.058768] init: ssh main process ended, respawning
Jul 31 22:46:19 hpclab kernel: [  253.948648] init: ssh main process (2987) terminated with status 255
Jul 31 22:46:19 hpclab kernel: [  253.948719] init: ssh main process ended, respawning
Jul 31 22:46:25 hpclab kernel: [  260.103095] init: ssh main process (2991) terminated with status 255
Jul 31 22:46:25 hpclab kernel: [  260.103174] init: ssh main process ended, respawning
Jul 31 22:46:28 hpclab kernel: [  263.077324] init: ssh main process (2995) terminated with status 255
Jul 31 22:46:28 hpclab kernel: [  263.077400] init: ssh main process ended, respawning
Jul 31 22:46:32 hpclab kernel: [  266.464170] init: ssh main process (2999) terminated with status 255
Jul 31 22:46:32 hpclab kernel: [  266.464238] init: ssh main process ended, respawning
Jul 31 22:46:35 hpclab kernel: [  270.124561] init: ssh main process (3003) terminated with status 255
Jul 31 22:46:35 hpclab kernel: [  270.124636] init: ssh main process ended, respawning
Jul 31 22:46:39 hpclab kernel: [  273.897050] init: ssh main process (3007) terminated with status 255
Jul 31 22:46:39 hpclab kernel: [  273.897124] init: ssh main process ended, respawning
Jul 31 22:46:43 hpclab kernel: [  277.403355] init: ssh main process (3011) terminated with status 255
Jul 31 22:46:43 hpclab kernel: [  277.403427] init: ssh main process ended, respawning
Jul 31 22:46:49 hpclab kernel: [  283.814673] init: ssh main process (3015) terminated with status 255
Jul 31 22:46:49 hpclab kernel: [  283.814745] init: ssh main process ended, respawning
Jul 31 22:46:52 hpclab kernel: [  286.418994] init: ssh main process (3019) terminated with status 255
Jul 31 22:46:52 hpclab kernel: [  286.419067] init: ssh main process ended, respawning
Jul 31 22:46:55 hpclab kernel: [  289.787432] init: ssh main process (3023) terminated with status 255
Jul 31 22:46:55 hpclab kernel: [  289.787506] init: ssh main process ended, respawning
Jul 31 22:46:58 hpclab kernel: [  293.078468] init: ssh main process (3031) terminated with status 255
Jul 31 22:46:58 hpclab kernel: [  293.078559] init: ssh main process ended, respawning
Jul 31 22:47:02 hpclab kernel: [  296.678693] init: ssh main process (3041) terminated with status 255
Jul 31 22:47:02 hpclab kernel: [  296.678773] init: ssh main process ended, respawning
Jul 31 22:47:07 hpclab kernel: [  301.814097] init: ssh main process (3045) terminated with status 255
Jul 31 22:47:07 hpclab kernel: [  301.814181] init: ssh main process ended, respawning
Jul 31 22:47:10 hpclab kernel: [  305.187638] init: ssh main process (3069) terminated with status 255
Jul 31 22:47:10 hpclab kernel: [  305.187718] init: ssh main process ended, respawning
Jul 31 22:47:14 hpclab kernel: [  308.759234] init: ssh main process (3074) terminated with status 255
Jul 31 22:47:14 hpclab kernel: [  308.759310] init: ssh main process ended, respawning
Jul 31 22:47:18 hpclab kernel: [  312.734980] init: ssh main process (3078) terminated with status 255
Jul 31 22:47:18 hpclab kernel: [  312.735052] init: ssh main process ended, respawning
Jul 31 22:47:22 hpclab kernel: [  316.840610] init: ssh main process (3082) terminated with status 255
Jul 31 22:47:22 hpclab kernel: [  316.840687] init: ssh main process ended, respawning
Jul 31 22:47:26 hpclab kernel: [  320.423302] init: ssh main process (3086) terminated with status 255
Jul 31 22:47:26 hpclab kernel: [  320.423377] init: ssh main process ended, respawning
Jul 31 22:47:29 hpclab kernel: [  323.789630] init: ssh main process (3090) terminated with status 255
Jul 31 22:47:29 hpclab kernel: [  323.789704] init: ssh main process ended, respawning
Jul 31 22:47:33 hpclab kernel: [  327.489133] init: ssh main process (3094) terminated with status 255
Jul 31 22:47:33 hpclab kernel: [  327.489201] init: ssh main process ended, respawning
Jul 31 22:47:36 hpclab kernel: [  331.205637] init: ssh main process (3098) terminated with status 255
Jul 31 22:47:36 hpclab kernel: [  331.205708] init: ssh main process ended, respawning
Jul 31 22:47:39 hpclab kernel: [  334.097717] init: ssh main process (3102) terminated with status 255
Jul 31 22:47:39 hpclab kernel: [  334.097784] init: ssh main process ended, respawning
Jul 31 22:47:43 hpclab kernel: [  337.910792] init: ssh main process (3106) terminated with status 255
Jul 31 22:47:43 hpclab kernel: [  337.910857] init: ssh main process ended, respawning
Jul 31 22:47:46 hpclab kernel: [  341.012934] init: ssh main process (3110) terminated with status 255
Jul 31 22:47:46 hpclab kernel: [  341.013008] init: ssh main process ended, respawning
Jul 31 22:47:49 hpclab kernel: [  344.211480] init: ssh main process (3114) terminated with status 255
Jul 31 22:47:49 hpclab kernel: [  344.211567] init: ssh main process ended, respawning
Any idea?
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Sendmail speedup rincewind Linux - Server 1 03-18-2007 08:26 PM
speedup download in linux boy_to_man Linux - Newbie 7 01-05-2007 03:58 AM
Linux boot up process Vs. Windows boot up process darkskull Linux - Software 7 12-30-2006 04:21 PM
speedup SUSE 10 bnjmn1989 SUSE / openSUSE 2 10-26-2005 03:51 PM
PostgreSQL , how to speedup ? ? kernelvn Programming 3 09-22-2004 11:43 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Ubuntu

All times are GMT -5. The time now is 04:26 PM.

Contact Us - Advertising Info - Rules - Privacy - LQ Merchandise - Donations - Contributing Member - LQ Sitemap -
Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration