[SOLVED] Scripting linux kernel compile

grail · 08-24-2010, 10:52 PM

Figured it must be my turn to run into a problem

I am trying to script the compilation and installation of a new kernel.
For most other applications the idea of the script below seems to be working fine.
The issue that has cropped up here is that whilst running the 'make menuconfig' step I am unable to use any of the arrow keys to navigate around

When I tracked down the cause (illustrated below) it appears to be due to my redirection for logging. So I am looking for advice on how best to redirect / log my scripts activity.
Reduced code that demonstrates the same issue (the assumption here is that kernel has been extracted and we are running the command from within the kernel directory):

Code:

#!/bin/bash
(
    make menuconfig
) 2>&1 | tee -a logfile

There are normally several other lines between the brackets '()', but running this illustrates my issue.

The enter key still seems to work, but arrows and tabs are all being sucked away by the redirection (I presume) and so I get things like:

Code:

^[[C

This is shown when pressing the right arrow button on the keyboard.
This is displayed on a black background in place of the first line chosen in the dialog interface.

Please let me know if any more details are required?

theNbomr · 08-24-2010, 11:41 PM

It isn't exactly clear what you're trying to accomplish, but I will guess that you want to capture the changes made to the config file. May I suggest simply saving a 'before' copy, and using diff to capture the changes? With the proper arguments, you can create a patch file, allowing you to re-create the changes.

--- rod.

crts · 08-24-2010, 11:56 PM

Hi grail,

I know the problem. I ran into it myself when I automated my LFS build. I am not 100% sure, however, I came to the conclusion that it does not really make sense to neither redirect stdout or pipe to tee in this case.
Think about it, stdout's content is in this case your menu. When you do a
cat logfile
after you abort with ctrl-c the output will be your previous screen because this was the content that was send to stdout. Do you really want to catch the menu?
One problem that the arrows do not 'register' is the pipe to tee, I think. You can verify this by
make menuconfig > logfile
Now (you might have to give the program some time first to make some configurations) hit the right arrow once (this selects exit) and then hit return. The program exits and you get your prompt back. So when you just redirect stdout the arrow does register with the program. You just don't see anything since output goes to 'logfile'.
Anyway, after realising that catching that blue configmenu into a file is not really necessary (you do have all the info you need in .config afterwards) I only redirected stderr to a file. Just in case something funny happens during config.

Hope this helps.

grail · 08-25-2010, 12:33 AM

Hey crts

Thanks for the information

You are a 100% that I do not wish to store any of the menuconfig interaction, except maybe that it was launched.

As you have probably guessed, this is part of my packaging system and whilst it works for all my other installs, because as yet none
of them have this type of interaction, it is only dying on this one

This part of the system captures all the steps from configure through to install and records both stderr and stdout so I can go back and see why an install
has errored and where the issue might be.

The redirect and tee was drawn from the LFS site for catching the log for either gcc or glibc. Is there a better way? Or do I need to find a way for this individual install
to tell the management not to redirect?

To give you more of an idea I have included below more steps as they are in my scripts:

Code:

(
    echo "Compile log for $SOURCE on $(date -u)"
    echo "Using gcc version: $(gcc -dumpversion)"

    pre_build_page       || report_error "pre build"
    build_page           || report_error "build"
    install_page         || report_error "install"
    post_install_page    || report_error "post install"
) 2>&1 | tee -a $C_LOG

Happy for any thoughts if you think this is not a good idea too

druuna · 08-25-2010, 02:20 AM

Hi grail,

Would this be of any help:

Code:

#!/bin/bash
#set -xv
set -u

kernelDir="/data/Downloads/linux-2.6.31.1"
logDir="/home/druuna"

doActions() {
  cd ${kernelDir}
  # make menuconfig
  make menuconfig
}

exec 2>&1

doActions | tee ${logDir}/kcompile.log

exit 0

This seem to work (all keys work when navigating the menuconfig and all is logged).

Hope this helps.

crts · 08-25-2010, 02:40 AM

Ok, here is an example how I handled such problems:

Code:

(
    echo "Compile log for $SOURCE on $(date -u)"
    echo "Using gcc version: $(gcc -dumpversion)"

    pre_build_page       || report_error "pre build"
) 2>&1 | tee -a $C_LOG

    echo "executing: make menuconfig" >> $C_LOG
    make menuconfig
    echo ".config file has been generated" >> $C_LOG

(
    build_page           || report_error "build"
    install_page         || report_error "install"
    post_install_page    || report_error "post install"
) 2>&1 | tee -a $C_LOG

Quote:

The redirect and tee was drawn from the LFS site

Where exactly from the LFS site? I used the 'build' template that came with the more_control_helpers package (user based package management). Here is an excerpt of it:

Code:

#!/bin/bash

configure_commands()
{ :
  ./configure --prefix=/usr
}

make_commands()
{ :
  make
}
...

{ configure_commands 3>&1 1>&2 2>&3 | tee "$HOME/configure.err" ;} &>"$HOME/configure.log"
...
{ make_commands 3>&1 1>&2 2>&3 | tee "$HOME/make.err" ;} &>"$HOME/make.log"
...

I prefer this since it generates a separate error.log. However, I do recall this thread that described a problem about mixed up order of messages. I think, it is because the threadstarter tried to redirect stdout from the calling script and in the called script itself at the same time. I have no explanation why this happens, i. e. why bash handles it this way. So keep in mind to avoid redirecting output while you call a script which redirects output itself.

Quote:

... yet none of them have this type of interaction ...

... except for the tzselect (glibc) and passwd (shadow) commands

I issued those at the end after everything was installed.

Quote:

this is part of my packaging system

IIRC, a modified 'user based' approach. You wanted to assign additional groups to certain packages to give them permission to make changes to files that belong to other packages, right? So is it working as expected? What problems did arise? As I have already stated on occasion I think user based package management is a very interesting technique. Unfortunately it is not very well explored, yet. So if you like to share, any input is appreciated.

crts · 08-25-2010, 03:04 AM

Quote:

This seem to work (all keys work when navigating the menuconfig

I can confirm that the arrow keys work. Which sort of disproves my previous assumption that the pipe to tee is the culprit. My guess was that tee somehow swallows the arrows before it throws the output back to the screen. Well, there goes the theory.

grail · 08-25-2010, 04:55 AM

Thanks a lot for the feedback guys

Will look into solutions tomorrow as my work machine is where I am hosting all this at the mo (under virtualbox)

Quote:

Originally Posted by crts

IIRC, a modified 'user based' approach. You wanted to assign additional groups to certain packages to give them permission to make changes to files that belong to other packages, right? So is it working as expected? What problems did arise? As I have already stated on occasion I think user based package management is a very interesting technique. Unfortunately it is not very well explored, yet. So if you like to share, any input is appreciated.

Good memory

Except for this small side track and that glibc seemed to have a heart attack today when being rebuilt (not sure why yet), all is going
quite well.

Issues - apart from revising the scripts to be more versatile (ie above problem with menuconfig), the two main introductions seem to be:

1. /usr/share - I have added an update_permissions function which looks through everything under this directory and sets group to install (main group) and permissions to 775. Reason is that as some things started to install things like /usr/share/locale/en/*, for example, when someone else has to update / add a file to the directory it could not.

2. Remove setuid - so far this was only on mount. As I am running upstart (natively) it is owned by upstart and it needed to be able to use things like mount

I can say the one thing that has made it really interesting is that now a simple ls -l tells you exactly what installed a particular file. So it saves a
lot of time when something breaks.

Another gotcha which I did run into, but this is more around my implementation rather than the actual packaging system, was that some programs cannot be removed prior to install. So my process for a normal install is currently the following phases, not all required for every application:

1. Extract and apply any necessary patches
2. PRE_CONF - perform any necessary seds, directory creations, etc needed for successful compilation
3. CONF - execute configure script with appropriate switches
4. MAKE - run make (sometimes extra steps required)
5. MAKE_CHK - make check and test for errors (if supported)
6. MAKE_INST - remove any folders / files previously installed that have attributes owner

wner (ie if owner:install then it will be left as other apps using it). Install application
7. POST_INST - add any final changes such as update config files, copy in new config files, move files / folders to be compliant

All of which is logged in compile and install logs.

Quote:

Where exactly from the LFS site?

http://www.linuxfromscratch.org/lfs/...r06/glibc.html

konsolebox · 08-25-2010, 04:58 AM

Hi. Back in 2008 I made a script for automating compilation of the kernel. It's no longer updated but I hope you get some idea from it.

http://www.linuxquestions.org/questi...script-632865/

grail · 08-25-2010, 05:29 AM

@konsolebox - will definitely have a look at what features i can pilfer

May ask what is the advantage of pushd / popd?

@konsolebox or druuna - this will show my ignorance a little here but could someone explain the line:

Code:

exec 2>&1 or exec 2>${errorfile}

I get the stderr part, but more along the lines of exec is doing here? Struggling a little to see how it follows the definition:

Code:

exec: exec [-cl] [-a name] [command [arguments ...]] [redirection ...]
    Replace the shell with the given command.
    
    Execute COMMAND, replacing this shell with the specified program.
    ARGUMENTS become the arguments to COMMAND.  If COMMAND is not specified,
    any redirections take effect in the current shell.

What COMMAND is being executed?

crts · 08-25-2010, 05:49 AM

Have a look here:
http://tldp.org/LDP/abs/html/x17601.html
exec can also be used for I/O redirection. Which is a more detailed explanation of the last sentence:

Quote:

If COMMAND is not specified,
any redirections take effect in the current shell.

So exec has apparently an inherent duality of usage.

konsolebox · 08-25-2010, 05:52 AM

Quote:

Originally Posted by grail

@konsolebox - will definitely have a look at what features i can pilfer

May ask what is the advantage of pushd / popd?

Nothing much really. After all you can use cd somewhere then cd - to go back.. maybe on multilevel operations.

@konsolebox or druuna - this will show my ignorance a little here but could someone explain the line:

Code:

exec 2>&1 or exec 2>${errorfile}

That command simply redirects the default output of fd 2 to somewhere else. So if you send output to fd 2, it will be redirected to the different output instead of stderr.

Quote:

Originally Posted by grail

I get the stderr part, but more along the lines of exec is doing here? Struggling a little to see how it follows the definition:

Code:

exec: exec [-cl] [-a name] [command [arguments ...]] [redirection ...]
    Replace the shell with the given command.
    
    Execute COMMAND, replacing this shell with the specified program.
    ARGUMENTS become the arguments to COMMAND.  If COMMAND is not specified,
    any redirections take effect in the current shell.

What COMMAND is being executed?

I don't really understand what you mean but.. there's none I guess.

Quote:

If COMMAND is not specified, any redirections take effect in the current shell.

grail · 08-25-2010, 06:06 AM

hmmm ... just goes to show I didn't look very hard

(Seeing as I recommend that site so many times

)

and it seems I can't even read my own copy / paste .. what drugs am i on (where can i get some more ... just jokes)

grail · 08-25-2010, 07:24 AM

Ok ... so I have been playing a little and just trying to find out if I am a victim of trying to be too clever or just missing something obvious.

So firstly, I just want to mention that I can guarantee that any script created will always be on bash 4+ (so not so concerned if it won't work elsewhere).

So I have my redirection as:

Code:

exec &> logfile

<do stuff here>

So I have two questions that have made me go round and round:

1. Should i wish to set a flag for the user so that the stderr/stdout is also redirected / directed to stdout, how would I go about this?

2. If I have things after the 'do stuff here' line that I do not wish to be in the logfile, how do I close the file off?

I looked at the command:

Code:

exec >&-

But of course this closes stdout and hence anything after that throws a wobbly

I was unable to find the equivalent for closing just the file redirection
(that of course could be due to more blindness as with all things you look at for too long)

catkin · 08-25-2010, 10:39 AM

Quote:

Originally Posted by grail

1. Should i wish to set a flag for the user so that the stderr/stdout is also redirected / directed to stdout, how would I go about this?

If I understood the question I might be able to help

Quote:

Originally Posted by grail

2. If I have things after the 'do stuff here' line that I do not wish to be in the logfile, how do I close the file off?

You can save the existing redirection by duplicating it to a spare file descriptor, do the things, restore the redirection and (for tidiness' sake) free the spare file descriptor.

Something like this (untested)

Code:

exec 3>&1   # Save stdout redirection on fd 3
exec 4>&2   # Save stderr redirection on fd 4
exec &> logfile
<do stuff here>
exec 1>&3   # Restore original stdout redirection
exec 2>&4   # Restore original stdout redirection
exec 3>&-   # Free fd 3
exec 4>&-   # Free fd 4
<do things>