I went from kernel 2.6.23.5 to 2.6.37 and init hangs before switching to run level 3. It executes all initial system scripts via "rc sysinit" at sysinit runlevel but never switches to run level 3 which is my initdefault. This was first seen with sysvinit-2.86 so I upgraded to sysvinit-2.88dsf and still I have the same issue.

I then enabled debug and added my own debug statements and found that it's hanging on select() inside check_init_fifo(). See code snippet here.

Why would select() hang when it has a five second timeout set?

Kernels 2.6.23.17 and 2.6.25.20 work fine. Kernels 2.6.35.10, 2.6.37, 2.6.38-rc1 all have the init hang issue.

I am compiling init on a 2.6.23.5 system so I don't know if that is an issue or not. The system has gcc 4.0.1 and glibc-2.3.5.

Is there a requirement to have glibc compiled against newer kernel headers?

Any help or insight would be much appreciated.

It is better to use the default gcc and glibc with distribution.

I've narrowed this down to init working properly in kernel 2.6.26.8 but not in 2.6.27 (two adjacent releases), at least not with their default configs (make defconfig). So then I copied the .config from 2.6.26.8 to 2.6.27, ran make, answered all questions with defaults, and booted the 2.6.27 kernel and now init is working properly (i.e., it's switching to runlevel 3 like it should).

So the next step is to figure out what kernel option is causing the problem. You can see the diff between the default non-working .config and the working .config here. If anyone has ideas what it might be, please share. In the meantime, I'll be going through and enabling/disabling different options to narrow it down.

I think I have it figured out. If this kernel option is NOT enabled, init hangs:

CONFIG_SYSFS_DEPRECATED=y
CONFIG_SYSFS_DEPRECATED_V2=y

Therefore, I believe the real problem is udev. I'm using an old version of udev that likely depends on deprecated features of sysfs. Updating udev should fix it. I don't think it's a coincidence that udev runs during sysinit runlevel before init switches to runlevel 3.