LinuxQuestions.org
Review your favorite Linux distribution.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 12-15-2010, 02:11 PM   #1
bennyt
Member
 
Registered: Dec 2010
Location: UK
Posts: 69

Rep: Reputation: 9
Kernel compilation and general new build frustration... why is it so tedious?


Hello all,

I've not posted before as I usually figure out problems eventually but my new build is driving me to madness. There is no specific problem, other than the usual ridiculous amount of effort that seems to accompany my infrequent builds.

So what I'd like to know is if I am missing something here.

I've been using Slackware for some time. Most recent build is on a Dell E4310 laptop, so I go for the latest version 13.1, 64 bit. etc. The bundled 'huge.s' kernel gets the thing going with plenty of hardware running so all seems good. This of course, is only the beginning because there is hardware not working, and the stock kernel is bloated Windows style, not optimised, and quite frankly an abomination.

So begins the seemingly horrendous task that always accompanies a kernel recompile. E.g. locating the right drivers in the kernel tree, if they exist, endless searching the web for support when it is not at all apparent what driver supports what bizarre new chipset or piece of hardware. There is apparently no way of interrogating the bloated kernel to find out what was compiled in to get it all running...

If one fails to identify the right drivers, or happens to miss just one crucial flag, the new kernel falls over. More rebuilding etc. I've been through the entire kernel tree attempting to chose the right options - probably 3 to 4 hours of my time. I have now had the new machine (Core i5 ~ 2.5GHz) compiling for an entire afternoon on a single kernel. It is still going even though I have removed most of the irrelevant options. Why is it so slow???? I swear my old P3 is faster, or have kernels just become fatter?

At any rate, I still do not have a new working kernel (though I am hoping this will be solved when it EVENTUALLY finishes building).

So, the question is, does this ring true with those of you who opt for a lean, optimised system, who take the time to recompile, build your own software sources etc, or am I missing something? While I am rusty at rebuilds, surely it should not take this amount of time and effort? (I estimate it will take at least 2 weeks of solid attention to have the thing vaguely working as I'd like. While I am a staunch Linux supporter it is a wonder that it has any supporters at all!).

Cheers,
Ben
 
Old 12-15-2010, 03:16 PM   #2
pljvaldez
LQ Guru
 
Registered: Dec 2005
Location: Somewhere on the String
Distribution: Debian Wheezy (x86)
Posts: 6,094

Rep: Reputation: 281Reputation: 281Reputation: 281
I used to rebuild kernels all the time and found it mostly painful in the same ways you do. Kernels have gotten fatter (which is the trade off for supporting every piece of hardware possible). Now I mostly just live with the stock kernels Debian ships unless there's a hardware problem (or it's a really old machine) and try to speed up my desktop by using light window managers and applications.

In the end, with kids and other stuff going on I just don't have the time to fiddle that I once did. So the time value outweighs the performance gain for me.
 
Old 12-15-2010, 03:58 PM   #3
mlangdn
Senior Member
 
Registered: Mar 2005
Location: Kentucky
Distribution: Slackware64-current
Posts: 1,845

Rep: Reputation: 452Reputation: 452Reputation: 452Reputation: 452Reputation: 452
Try Slackware's generic kernel. There is a README.initrd in /boot that gives a good how-to on implementing this kernel.
 
Old 12-15-2010, 06:31 PM   #4
bennyt
Member
 
Registered: Dec 2010
Location: UK
Posts: 69

Original Poster
Rep: Reputation: 9
Hi, thanks for the replys. (Rant ensues ) It always seems to come down to a balance of 'performance' (or rather perfection!) and time. I can't help feeling that I should just accept the bundled option and not worry about it but then if that's the case, why not just pay a Windows license fee and get used to it? To be fair, Windows 7 was installed as standard and everything worked perfectly and fairly swiftly (albeit on a powerful dual core with 8GB, and taking a whopping ~35GB for the OS!!). One can always ssh into another platform.

Trouble is, for me part of the attraction of *nix is that one is trying to get efficient performance from hardware, and the very fact that customisation is itself demonstration of understanding - as opposed to the whole install and go (don't understand but pay the man) philosophy of other platforms. (Not to mention the hopelessness of performing certain tasks under big brand OSs).

Without taking a pop at developers or the community (to which I contrbiute very little and who do a great job), as far as Linux has come in terms of becoming 'accessible', one of the fundamental points and advantages, for me at least, is lost and unrealistically achievable for all but the accomplished. That is, I would like to have an OS and kernel that is built for my system and does precisely what <I> want it to, efficiently, and is not limited (crippled) by a single developer's ideas (or clandestined marketing schemes and back-handers with other massive corporations - yes believe it or not you do not need a quad core goliath and 40GB of disk space to run a web brower and word processor).

However, so that I do not come across as a total whingebag I shall write up a piece on installing Linux on the E4310 for others that are forced to buy it (its OK, and powerful for a small laptop but the screen is rubbish, and its a Dell) - drawing attention in more general terms as to how to rebuild the kernel (I don't mean the commands to execute, I mean the general steps and tedious process in determining hardware, drivers, how to get everything working, and the order in which to tackle certain problems).

Sorry I'm preaching to the masses of course but I know there are a lot of newbies on here who probably have one go at Linux and think wtf is this madness - but I think they have every right to in a lot of cases.

Cheers,
 
Old 12-15-2010, 07:21 PM   #5
mlangdn
Senior Member
 
Registered: Mar 2005
Location: Kentucky
Distribution: Slackware64-current
Posts: 1,845

Rep: Reputation: 452Reputation: 452Reputation: 452Reputation: 452Reputation: 452
Up until early this year, I always did a custom kernel as the first thing to do after a clean install. I have not done that since spring. The generic works very well. I think my custom kernels were faster, but what's a few milliseconds, or even a couple seconds of speed in booting really going to do for me? For the record, I think I'm going to build a custom kernel next week, just so that I don't get rusty.
 
Old 12-15-2010, 09:36 PM   #6
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,578
Blog Entries: 31

Rep: Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208
Quote:
Originally Posted by bennyt View Post
Most recent build is on a Dell E4310 laptop, so I go for the latest version 13.1, 64 bit. etc. The bundled 'huge.s' kernel gets the thing going with plenty of hardware running so all seems good. This of course, is only the beginning because there is hardware not working, and the stock kernel is bloated Windows style, not optimised, and quite frankly an abomination.

So begins the seemingly horrendous task that always accompanies a kernel recompile. E.g. locating the right drivers in the kernel tree, if they exist, endless searching the web for support when it is not at all apparent what driver supports what bizarre new chipset or piece of hardware. There is apparently no way of interrogating the bloated kernel to find out what was compiled in to get it all running...

If one fails to identify the right drivers, or happens to miss just one crucial flag, the new kernel falls over. More rebuilding etc. I've been through the entire kernel tree attempting to chose the right options - probably 3 to 4 hours of my time. I have now had the new machine (Core i5 ~ 2.5GHz) compiling for an entire afternoon on a single kernel. It is still going even though I have removed most of the irrelevant options. Why is it so slow???? I swear my old P3 is faster, or have kernels just become fatter?

...

So, the question is, does this ring true with those of you who opt for a lean, optimised system, who take the time to recompile, build your own software sources etc, or am I missing something? While I am rusty at rebuilds, surely it should not take this amount of time and effort?
Laptops, netbooks etc. have more special hardware than desktops and servers so need more special drivers. Generally they come with a custom version of Windows on which the OEM has already done the sort of work you are doing. Installing standard Windows on these machines usually requires hunting for and installing extra drivers.

huge.s is a compromise between supporting all possible hardware (when it would be mega-gargantuan.s) and size; for you it is already "bloated" and "an abomination" but any reduction in size would require more people to build custom kernels to support their hardware.

When building a custom kernel, unless somebody has kindly posted a .config file for your model or similar, you have two choices -- to start with one of the existing kernels or to start with a default configuration. I found it very much easier to start with an existing configuration and take things away than with the default and add because there are some strange defaults IMHO. As long as your kernel is configured for it (ha!) you can get a running kernel's configuration by zcat /proc/config.gz > .config_from_running_huge_kernel. If that doesn't work, consider changing distros because your present distro is not designed for people like you!

Your kernel build time sounds excessive. Mine take ~30 minutes (without specifying a -j value) using an Athlon x2 Dual Core BE-2300, a processor chosen for its low power consumption. Even allowing for laptop drives being chosen for low power consumption the time you report is long (I have a Western Digital WD3200AAKS-0063A0, again chosen for low power consumption but it is faster than most laptop drives). The task is CPU-intensive so it may be that your CPU is getting hot and the system is slowing it down; extra cooling could help.

Yes, this rings true. My kernel configuration log is 56 pages long. It records the building of 6 kernels and there are still things I would like to remove but doing so is now low on the priority list. It is so long because I copy and paste from the menuconfig screens to give an exact record (errors and omissions excluded) of the changes made so when something breaks I can selectively undo changes until it works again and start over.

Given the complexity of the kernel and the many options available, the process cannot be much simpler.
Quote:
Originally Posted by bennyt View Post
It always seems to come down to a balance of 'performance' (or rather perfection!) and time. I can't help feeling that I should just accept the bundled option and not worry about it but then if that's the case, why not just pay a Windows license fee and get used to it?

...

Trouble is, for me part of the attraction of *nix is that one is trying to get efficient performance from hardware, and the very fact that customisation is itself demonstration of understanding ...

...

I shall write up a piece on installing Linux on the E4310 for others ...
The majority of people do simply run a stock kernel and, of course, there's a lot more to the Linux/Windows differences than kernel size! Comparing the costs and benefits of stock kernels and creating custom kernels I conclude that we only build custom kernels for the satisfaction and challenge; you would have to boot many times and sit waiting for it to finish before the hours (or days!) of custom kernel building are repaid.

Giving that work back to the community increases the total benefit deriving from the work done -- and so justifies it if justification is needed. In the case of a widely available model like yours, many could benefit. If you would like to see my kernel build log an maybe adapt its format to your own purposes, please ask. It's an OOo Writer file.
 
Old 12-16-2010, 02:48 AM   #7
H_TeXMeX_H
LQ Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301
You don't have to build a custom kernel, you can use generic + initrd. When building a kernel you should also decide how much time you want to put into it. I recommend just going through the processor section and optimizing for your processor and latency options if you want a quick custom build. If you want to spend lots of time customizing, you can, but the benefit will not scale linearly.
 
Old 12-16-2010, 07:07 AM   #8
bennyt
Member
 
Registered: Dec 2010
Location: UK
Posts: 69

Original Poster
Rep: Reputation: 9
Hi all, and thanks again.

Catkin, thanks, please do ping me copy of your build log - can you do this via my contact options (not sure how this forum works)?

I shall investigate a few of the those tips. While the kernel build is, as generally pointed out, only likely to improve boot times (fairly useless) and perhaps have a minor impact on general performance, there is still the issue of hardware not working intially, especially for laptops with bits like bluetooth, webcams, mics and other more unusual features.

My kernel did rebuild and run when I included the right sata driver (AHCI sata?) so now I can look forward to more customisation, with software to follow.

For the record, a few things I've come across:
i) On running make xconfig, it fell over complaining QT3 and libraries didn't exist. (A new full install slack 13.1 and I can't run make xconfig, that never happened before...)
ii) Menuconfig failed to save my options so I had to save it to a custom file and copy it to .config manually (why?).
iii) On creating a clean build, the default appears to be to select everything?? This might(?) easily be changed but why does the default not reflect somewhere around the middle of the distribution of users and applications? (i.e. say infiniband support is unlikely to be applicable to most users...)
iv) Slow compile time so testing has been enormously time consuming.

And I am now finding that xorg.conf has been scrapped and I have to work with hal... omg?? lol (Xfreezes instantly - I foresee a new thread in the slackware forum)

Thanks for the support and I shall press on.
Cheers,
Ben
 
Old 12-16-2010, 12:46 PM   #9
mlangdn
Senior Member
 
Registered: Mar 2005
Location: Kentucky
Distribution: Slackware64-current
Posts: 1,845

Rep: Reputation: 452Reputation: 452Reputation: 452Reputation: 452Reputation: 452
For xconfig to work, you must install the qt-3x libs from the Slackware install disk (or your local tree) /extra/kde3-compat
 
Old 12-16-2010, 02:47 PM   #10
jiml8
Senior Member
 
Registered: Sep 2003
Posts: 3,171

Rep: Reputation: 116Reputation: 116
There are a golzillion different possible hardware configurations (hint: Golzillion is greater than a zillion and less than a bazillion) so having a generic kernel that works with pretty much everything is lots easier than tailoring a kernel to be lean, mean, and only work with the hardware that is present.

I don't know about you, but I find hard drive space, RAM, and CPU cycles to be cheap these days, while my time is scarce and very valuable. So going with stock kernels works for me.

That said, I am hacking a kernel right now. But it is for an embedded device, so I have to pare it way down. But for my working machines...who gives a rat's patootie.

A complete kernel build for 64 bit OpenSUSE 11.3 running as a VMware Workstation virtual machine on a Phenom-II 955 takes about 40 minutes, when I build without the -j option.

Last edited by jiml8; 12-16-2010 at 03:13 PM.
 
Old 12-16-2010, 09:45 PM   #11
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,578
Blog Entries: 31

Rep: Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208
Quote:
Originally Posted by bennyt View Post
ii) Menuconfig failed to save my options so I had to save it to a custom file and copy it to .config manually (why?).
iii) On creating a clean build, the default appears to be to select everything?? This might(?) easily be changed but why does the default not reflect somewhere around the middle of the distribution of users and applications? (i.e. say infiniband support is unlikely to be applicable to most users...)
ii) Never had that problem. Do write periodically when making many changes because an accidental Ctrl+C looses changes.
iii) After make mrproper I found the .config file had very few things selected.

Kernel build log attached. Change .pdf to .odt
Attached Files
File Type: pdf CW8 Slackware64 13.1 kernels log.pdf (88.2 KB, 14 views)
 
Old 12-16-2010, 10:18 PM   #12
Kenny_Strawn
Senior Member
 
Registered: Feb 2010
Location: /usa/ca/orange_county/lake_forest
Distribution: ArchBang, Google Android 2.1 + Motoblur (on Motortola Flipside), Google Chrome OS (on Cr-48)
Posts: 1,791
Blog Entries: 62

Rep: Reputation: 56
Quote:
Originally Posted by catkin View Post
ii) Never had that problem. Do write periodically when making many changes because an accidental Ctrl+C looses changes.
iii) After make mrproper I found the .config file had very few things selected.

Kernel build log attached. Change .pdf to .odt
You do realize you can still view PDFs in Linux, right? There's Evince on GNOME and Okular on KDE that you can use just as easily as OpenOffice or LibreOffice.
 
Old 12-17-2010, 05:49 AM   #13
mlangdn
Senior Member
 
Registered: Mar 2005
Location: Kentucky
Distribution: Slackware64-current
Posts: 1,845

Rep: Reputation: 452Reputation: 452Reputation: 452Reputation: 452Reputation: 452
Quote:
Originally Posted by Kenny_Strawn View Post
You do realize you can still view PDFs in Linux, right? There's Evince on GNOME and Okular on KDE that you can use just as easily as OpenOffice or LibreOffice.
odt is not a valid extension for upload, pdf is, so that's why he probably changed the extension to pdf.
 
Old 12-17-2010, 05:57 AM   #14
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,578
Blog Entries: 31

Rep: Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208
Quote:
Originally Posted by mlangdn View Post
odt is not a valid extension for upload, pdf is, so that's why he probably changed the extension to pdf.
Hole in one!
 
Old 12-28-2010, 06:57 PM   #15
bennyt
Member
 
Registered: Dec 2010
Location: UK
Posts: 69

Original Poster
Rep: Reputation: 9
I'll dig this up for completeness - and thanks for the replies (and build log from Catkin) - I had not forgotten (entirely!).

Here is the build summary (of sorts) for the E4310, as promised.
http://www.linuxquestions.org/questi...-e4310-852914/

Unfortunately the sound and graphics required a newer kernel so a recompile turned out to be unavoidable. I'm definitely still making a mess of my kernel build since the last compile time for kernel and modules was about 4 hours. However, it works and I desperately need to get some real work done

Cheers.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] How to redirect compilation/build error to a file? archieval Linux - Kernel 2 09-09-2010 02:27 AM
Virtualbox-ose compilation error using Sbo slack-build samac Slackware 11 10-30-2008 02:28 PM
Which is the kernel build directory? I Can't find kernel build files crabsody Debian 7 01-09-2007 04:48 AM
Tedious migration process??? zwdaniel Linux - General 2 04-27-2005 03:57 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 02:18 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration