LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Slackware (https://www.linuxquestions.org/questions/slackware-14/)
-   -   If I were to recompile & arch-optimize Slackware, where to begin? (https://www.linuxquestions.org/questions/slackware-14/if-i-were-to-recompile-and-arch-optimize-slackware-where-to-begin-558415/)

jong357 06-06-2007 11:39 AM

Quote:

Originally Posted by GrapefruiTgirl
1 - pkgtool doesn't like tar 1.16 so to use pkgtool I had to install tar 1.13. trouble was, in CHROOT, tar 1.13 didn't seem able to unpack bzip2 archives

Do this:
Code:

echo $PATH
/temptools/bin should be at the very end. That's your toolchain and bzip was built in the toolchain section. If it's not in your PATH then your not re-entering back into your system right. Oh... Your using tar-1.13... See below. Maybe tar-1.13 doesn't like bzip-1.0.4

Quote:

I reinstalled 1.16, but it wrote over 1.13 :( so I had to reinstall 1.13, and then install 1.16 as "/bin/tar116" and use it specifically if/when I need it. I install pkgtools by hand, so had to also install 'dialog' and build & install 'mktemp' as well
Go back and read my pkgtools build script. The Slackware build script for tar works the same.
Code:

install -m 755 src/tar                $PKG/bin/tar-1.13
pkgtools is hard coded to look for a binary called "tar-1.13". If it doesn't find one, it'll use "tar". Your regualar tar-1.16 should be named "tar" and tar-1.13 for pkgtools should be named "tar-1.13". You want it to be the opposite of how it is now. Calling on just 'tar' should use our new/updated tar-1.16 and the only reason 'tar-1.13' should get used is by pkgtools.

Quote:

I had the PM_DEST short one level, so the damned headers kept installing to /include rather than /usr/include
Sometimes it's better to just copy and paste to avoid typing errors... Another reason to be using scripts.
Code:

make headers_install INSTALL_HDR_PATH=$PM_DEST/usr
Good to see you got a good glibc build. I'm curious, are you still using the DIY /etc/group? Do you know why your not getting any glibc errors now?

Am I right in assuming your making packages out of everything now? If not, then the above command would have been:

Code:

make headers_install INSTALL_HDR_PATH=/usr

GrapefruiTgirl 06-06-2007 11:45 AM

Yep indeed :) making packages out of EVERYTHING :)

The tar situation was indeed weird for a bit. I had it set up the same as my real system, which ahs both, but it didn't work. And I checked the path; it WAS in there, yet it still wouldn't work. I finally installed it from scratch, from within the chroot, using installpkg, and it worked :) but I still have tar-1.16 installed as "tar116' so I can call it when I need it.
Correct: Tar 1.13 doesn't like bzips.

Also, it isn't necessarily typos on my part; it's muchly a copy & paste process, however, the occasional cp -v command to 'install' stuff to /tmp the way they do in the docs, doesn't create the parent directories, so I had to send just the final part of the destination path to /tmp, then go there and create the parents and put the stuff in order. So while learning how to make the packages (which I had never done manually with 'makepkg' before) I was repeatedly sitting in the wrong folder when making the packages :p but all good now :)

All is well so far, but it looks like it'll be a battle with GCC now for some reason. The first run of torture tests didn't produce very good results, so I just rebooted and am starting it again right now..

jong357 06-06-2007 12:17 PM

Quote:

Originally Posted by GrapefruiTgirl
Yep indeed :) making packages out of EVERYTHING :)

Good! Next time around, use scripts. You'll thank yourself for it... ;) It's MUCH easier.
Code:

sh glibc.build
installpkg /tmp/glibc-2.4-i686-1.tgz

Quote:

The tar situation was indeed weird for a bit. I had it set up the same as my real system, which ahs both, but it didn't work. And I checked the path; it WAS in there, yet it still wouldn't work. I finally installed it from scratch, from within the chroot, using installpkg, and it worked :) but I still have tar-1.16 installed as "tar116' so I can call it when I need it.
If I'm not mistaken, Slackware installs tar-1.13 as "tar-1.13" and installs tar-1.16 as "tar-1.16" and then symlinks 'tar-1.16' to 'tar'. That way when 'tar' is called, it uses tar-1.16... Make sure you do have a plain old "tar" in your path (that's not tar-1.13)...

Quote:

however, the occasional cp -v command to 'install' stuff to /tmp the way they do in the docs, doesn't create the parent directories, so I had to send just the final part of the destination path to /tmp, then go there and create the parents and put the stuff in order.
Ewww.... :p What a pain. Make sure you get everything in the right place. Another reason to use scripts I guess.
Code:

mkdir -p $PKG/{etc,bin,usr/man/man1}
install -m 755 dialog                        $PKG/bin
install -m 644 dialog.1                        $PKG/usr/man/man1
install -m 644 samples/slackware.rc        $PKG/etc/dialogrc

Which reminds me, have you actually tried running "pkgtool" yet?

Quote:

I was repeatedly sitting in the wrong folder when making the packages :p but all good now :)
Yep. Make sure you are in the root of $PM_DEST before running 'makepkg'

Quote:

The first run of torture tests didn't produce very good results, so I just rebooted and am starting it again right now..
Make sure you compare your results with the ones listed at DIY. I'm sure you are. The gcc tests aren't as 'critical' as the glibc tests. It's not uncommon to get some "unexpected failures" so don't sweat it if you do. As long as they are "close" to what greg posts on the GCC lists, then package it up. It'll work...

Oh... I guess it doesn't matter too much but glibc is a wondefull package to optimize at -O3... That's one of the reasons your doing this, no? I'd set your -O2 package in a safe place and go back and rebuild it at -O3 if I were you. But that's just me... ;) But... You've probably already re-adjusted your toolchain so that makes things a little more complicated. A little but not really.

gnashley 06-06-2007 12:50 PM

Once you are compiling stuff with the new compiler and want to make packages you should be able to use src2pkg without any trouble. The package installer currently uses the Slackware installpkg though I have already been looking at the tar-compatibility problems.
The problem revolves around the -l option which has been changed and the --no-overwrite-dirs option which was the standard behaviour before. The -l option has changed so the long syntax is needed --one-file-system. A small change to installpkg should make it work the same with newer tar versions or tar-1.13, something like this:

Code:

--- ./installpkg.00        2004-05-30 02:20:34.000000000 +0200
+++ ./installpkg        2007-06-06 18:25:01.000000000 +0200
@@ -416,7 +416,11 @@
    cat $DESCRIPTION | grep "^$shortname:" >> $ADM_DIR/packages/$shortname 2> /dev/null
  fi
  echo "FILE LIST:" >> $ADM_DIR/packages/$shortname
+  if [ "$TAR" = "tar-1.13"] ; then
  ( cd $ROOT/ ; $TAR -xzlUpvf - ) < $package >> $TMP/$shortname 2> /dev/null
+  else
+  ( cd $ROOT/ ; $TAR --no-overwrite-dirs --one-file-system -xzUpvf - ) < $package >> $TMP/$shortname 2> /dev/null
+  fi
  if [ "`cat $TMP/$shortname | grep '^./' | wc -l | tr -d ' '`" = "1" ]; then
    # Good.  We have a package that meets the Slackware spec.
    cat $TMP/$shortname >> $ADM_DIR/packages/$shortname

Be sure to grab the latest src2pkg-1.2 which was never announced, though I am working on a new update the last few days -some better docs and also exactly what you are wanting to do with scripting the build.
Starting with 1.2 we have support for downloading multiple sources associated with a build, even starting the whole thing off from the command-line with a simple 'src2pkg -x URL-address-of-src2pkg-script'

I use a very simple GroupBuild script for building groups of packages, whether they depend on each other or not. You can change the behaviour pretty easily depending on what you wanna do:

Code:

#!/bin/bash
THISDIR=$(pwd)
EXEC_NAME="${0##*/}"

# This is used to build repo-style -comment out to use default dirs
# sources are in a directory relative to this one
export SOURCES_DIR=`pwd`/../../Sources/g
# place packages in a common directory relative to this one
export PKG_DEST_DIR=`pwd`/../../Packages
# if you need to include otjer options common to all the packages -like for gnome
# export EXTRA_CONFIGS='--sysconfdir=/var'

# -I is used for chain builds where packages need to be installed
# -W will cleanup -only use this when you are sure everything builds cleanly
# -X use -X if you are running a list of src2pkg scripts in directories
TRANSIENT_OPTIONS="-I -W -X"

# BUILD_LIST can be a list of directories in the THISDIR
BUILD_LIST="$(ls $THISDIR)"
# or use a fixed list if you need to specify the list or order
# BUILD_LIST="libidl atk"

echo "Running $EXEC_NAME in $THISDIR"

for dir in $BUILD_LIST ; do
        cd $THISDIR
        if [[ -d $dir ]] ; then
        # ignore non-directories in THISDIR
        cd $dir
       
        # Assuming that each directory contains a .src2pkg build script
        echo "Running src2pkg $TRANSIENT_OPTIONS in $dir"
        src2pkg $TRANSIENT_OPTIONS
       
        fi

done

This includes no error checking and assumes that each build and the build order is working correctly.

You'll of course want to compile your src2pkg from scratch since you are building a new system -especially since the new version uses newer installwatch libs.
Get the latest package or sources at src2pkg's new home here:
http://distro.ibiblio.org/pub/linux/...nload/src2pkg/

GrapefruiTgirl 06-06-2007 01:18 PM

Gnashley -- thank you very much for that reply above :)

Ummm -- jong357, I need to clarify something here (or gnashley or anyone who's here who knows for sure) regarding making the packages.. I know we have covered this, but something just isn't right here, and I think it's glibc's installation; I have literally thousands of GCC test-suite errors..
Here's a simple example: Let's say I want to make a package which is entirely comprised of one file, in one folder, like so:
Code:

/ETC
  |_____
        |
    file.conf

OK? so this is the whole package: one config file that ultimately gets installed into /etc.
My $PM_DEST is "/tmp" so after running the install, I end up with /tmp/etc/file.conf.
Now, when I run 'makepkg', should I be sitting in /tmp or sitting in /tmp/etc ??
Based on your above post, I figure I should be in /tmp but please clarify.

Thanks so much. :)

jong357 06-06-2007 01:26 PM

Yea, sounds like you need to get ahold of making packages. To answer your question directly, you need to be in /tmp, not /tmp/etc

1.) Don't use /tmp for $PM_DEST. Use /tmp/package-NAMEOFPACKAGE
2.) cd into /tmp/package-NAMEOFPACKAGE
3.) makepkg /tmp/NAMEOFPACKAGE-VERSION-ARCH-BUILDNUMBER.tgz
4.) installpkg /tmp/NAMEOFPACKAGE-VERSION-ARCH-BUILDNUMBER.tgz

You need to be in the ROOT of the package before issuing 'makepkg'. Your probably also not making an install/slack-desc file either, right? Not overly important to the sanity of the package but when you install it, you won't see any sort of description...

ALWAYS use a seperate CLEAN directory for each package. If you don't (and don't rm -rf on everything in that directory for each package), you'll progressively get larger and larger packages as you go. Before you know it, you'll have a 500mb package that also contains all the previous packages before it.

And yes, you've probably borked your glibc package by not packaging it properly. You should have seen some problems with the bottom commands of "Readjust Toolchain" if your new glibc wasn't right. What was the output from those sanity checks?

You should have seen this:
Code:

[root@jaguar ~] # 1. dynamic linker
[root@jaguar ~] readelf -l a.out | grep ": /lib"
      [Requesting program interpreter: /lib/ld-linux.so.2]
[root@jaguar ~]
[root@jaguar ~] # 2. start files
[root@jaguar ~] grep " /usr/lib/crt1.o succ" foo
attempt to open /usr/lib/crt1.o succeeded
[root@jaguar ~]
[root@jaguar ~] # 3. libc
[root@jaguar ~] grep " /lib.*libc.so.6 succ" foo
attempt to open /lib/libc.so.6 succeeded
[root@jaguar ~]
[root@jaguar ~] # 4. deps found from DT_NEEDED
[root@jaguar ~] grep "^found.*at /lib.*/ld" foo
found ld-linux.so.2 at /lib/ld-linux.so.2

And NOT /temptools stuff. That's why we run those sanity checks to make sure we are using our new toolchain and not the /temptools one, and also to verify that it's working correctly.

Not a big deal. Do a "removepkg" on everything thus far and start over with man-pages. This time try using -O3 on glibc... :-) It also wouldn't hurt to use -march=i686 for everything too... (if that is indeed your processor ARCH)

GrapefruiTgirl 06-06-2007 03:28 PM

OK, as per your above post, I have the package-making part correct as far as I can see.. (Remember I used /tmp as my PM_DEST) I did:
Code:

make install $PM_DEST/some-package-name
cd /tmp/some-package-name
makepkg /tmp/package-1.2.3-i686-1.tgz
cd /tmp
installpkg package-1.2.3-i686-1.tgz

so this appears to be OK... However I am going to do it EXACTLY as you posted above, this time.

But something's screwed up.

The sanity checks returned ALL OK. Everything was found and identified correctly as you wrote above.

I don't think I have the compounded-archive problem from not cleaning out the tmp dir each time, because I actually made the package from the contents of the first folder, and not from /tmp, however I can't be sure, so I will remove all the packages and go back to glibc. should work fine this time round, and will be worth making sure.

I didn't use -O3 because of everything I have read: while it is actually fine to optimize glibc, the general concensus seems to be that O3 does not necessarily make an installation any faster, but often makes it slower. Perl is done at O3 because that's what the Perl people intended and tested, but since I have done the entire system at "-O2 -march=i686 -mtune=pentium4 -pipe" I figured I'd stay on the safe side since it actually compiled fine; remember, prior to that, it was at -O3 as perthe instructions, but I backed it down to O2 and it compiled.

I will do it again though. I want this perfect :) and if O3 works the first time, Ill use it --- if it fails but works at -O2, then I'll go with -O2.

And yes -- that's basically the purpose of all this: the CC optimizing, besides the upgrades :) so I have done the -686 optimizing.

OK, back in a bit!

jong357 06-06-2007 05:39 PM

Quote:

Originally Posted by GrapefruiTgirl
OK, as per your above post, I have the package-making part correct as far as I can see.. (Remember I used /tmp as my PM_DEST) I did:
Code:

make install $PM_DEST/some-package-name
cd /tmp/some-package-name
makepkg /tmp/package-1.2.3-i686-1.tgz
cd /tmp
installpkg package-1.2.3-i686-1.tgz

so this appears to be OK...

Well, I'm sure you just left it out, but you need to be doing this:
Code:

make install DESTDIR=$PM_DEST/some-package-name
Don't forget the DESTDIR bit otherwise your just doing a straight up "make install". Glibc uses "install_root" and others use "prefix" but for the most part, it's DESTDIR...

Also, you should just define $PM_DEST as "/tmp/package-PACKAGENAME". So for glibc, it would be:
Code:

PM_DEST=/tmp/package-glibc
make install_root=$PM_DEST install -j1

Or, if you don't like continually defining variables by hand:
Code:

make install_root=/tmp/package-glibc install -j1
Quote:

I didn't use -O3 because of everything I have read: while it is actually fine to optimize glibc, the general concensus seems to be that O3 does not necessarily make an installation any faster, but often makes it slower.
Complete and utter rubbish.. ;) Go ask Roland McGrath and Ulrich Dreeper what they think about it. If anybody knows what they are talking about when it comes to glibc, it's the glibc devs. But, you are always ofcourse welcome to build your system as you see fit. Not really a big deal, all and all (I just don't like seing FUD :-). Sounds like something you might read on LFS... :-)

Quote:

remember, prior to that, it was at -O3 as perthe instructions, but I backed it down to O2 and it compiled.
Well, your associating there without any real basis in fact. I build glibc all the time with -O3 and it works just fine with no errors, so there would have to be something else going on. Now, it is quite possible when you start defining multiple OPTS that something could go wrong. -mtune is really semi-pointless IMO and in fact will make gcc-3.3.6 bomb (think cxxlibs). Building glibc with "-march=i686 -pipe -O3" is perfectly acceptable. Anyhoo...

Good luck... Your a trooper, that's for sure. ;)

jong357 06-06-2007 06:06 PM

A good way to see what the glibc devs intend is to look at the fedora spec file for glibc. Both those guys work for redhat.

http://cvs.fedora.redhat.com/viewcvs...ec?view=markup
Quote:

%ifarch i386
%define nptl_target_cpu i486
%else
%define nptl_target_cpu %{_target_cpu}
%endif
%ifarch %{ix86}
BuildFlags="-march=%{nptl_target_cpu} -mtune=generic"
%endif
%ifarch i686
BuildFlags="-march=i686 -mtune=generic"
%endif

build_CFLAGS="$BuildFlags -g -O3 $*"
They usually build for the lowest common denominator so, it would be (for glibc-2.6):

build_CFLAGS="-march=i486 -mtune=generic -g -O3"

GrapefruiTgirl 06-06-2007 06:07 PM

:) here I go again -- fresh! You're right, I had the DESTDIR= either missing or otherwise screwed up. Plus, somehow /libexec got installed to / rather than /usr which is why the entire GCC test-suite took an extra long lunch break -- and never returned :p ; I suspect, it is because I screwed up the package making.
I'll be keeping this thread handy when I get to CHROOT again.

RE: Someone asked above if I tried src2pkg yet.. I did, but it didn't work because of the problems I just covered.

Have I scripted this yet? No, because I learn more by typing stuff. Pressing GO doesn't teach anyone anything, and besides, with all the sed's and other tweaks and stuff to the makefiles and all, the script, when finally ironed out, would only be good for THIS EXACT COLLECTION of packages. Sure, it might save me a little while during this learning phase, but in the long run, I'd rather have gone thru it by hand, because I pick up new things every run.

I will go with -O3 for the glibc again this time, but what ebout "everything" else on the system? What's your opinion of -O3 for stuff in general? I have read quite a bit that claims it is often counterproductive. I have so far done everything on -O2, and will this time too, except glibc, but I'd appreciate you opinion..

OK, here I go. Back soon.

jong357 06-06-2007 06:40 PM

Quote:

Originally Posted by GrapefruiTgirl
somehow /libexec got installed to / rather than /usr which is why the entire GCC test-suite took an extra long lunch break -- and never returned :p

Well, unless you've modified the ./configure commands, DIY doesn't use /usr/libexec but instead places them in /usr/lib...

But yea... You've got to nail down the package making thing first off. Otherwise nothing is where it's supposed to be... ;)


Quote:

scripts, when finally ironed out, would only be good for THIS EXACT COLLECTION of packages.
Not true. That's why you make conditionals. Look at my gcc build script and DIY's gcc build commands. Nothing like the good old if/then kind of statement... Or if you really want to see a tangled mess of if/then's, look at my python script. You never know what kind of python package you'll wind up with. It just depends on what's on your system at the time...

Quote:

What's your opinion of -O3 for stuff in general?
probably not a good idea... :D Or rather not worth it in most cases. In situations like binutils (2.17 FSF) it will crap out on -O3.. I never used to have problems with -O3 for years but I've been seeing occasional breakage here and there with certain progs. -O2 is perfectly fine, generally speaking. You can use -O3 if you want, just remember to unset that bit when you do binutils-2.17.. Not sure if that's an issue with the HJL release or not. but with FSF 2.17, don't use -O3.

Quote:

I have so far done everything on -O2, and will this time too, except glibc,
Well, I wasn't saying that you shouldn't use -O2 for glibc, I was just saying that statement was a load of FUD is all... :p -O2 is fine for glibc as well. Just depends on what you want to do. Just don't belive everything you read (especially if it comes from LFS :D Kiddin. For the most part....)

H_TeXMeX_H 06-06-2007 06:58 PM

Best thing to use, I think, is what you get when you compile a kernel. For example I get:

Code:

-O2 -fomit-frame-pointer -pipe -march=i686 -mtune=pentium4
That's for a pentium4, and technically, you only really need -march=pentium4 and no -mtune.

I strongly recommend against -O3. It breaks SO many programs ... it's just not worth it, IMO.

jong357 06-06-2007 07:29 PM

The "external links" at the bottom has some good info

http://en.wikipedia.org/wiki/CFLAGS

H_TeXMeX_H 06-06-2007 09:19 PM

Good, here's a nice Gentoo wiki on safe cflags:
http://gentoo-wiki.com/Safe_Cflags

One more thing you might wanna know (if you don't already). As you know the standard Slackware CFLAGS are:
Code:

-O2 -march=i486 -mtune=i686
This means
Quote:

Slackware Linux can run on 486 systems all the way up to the latest x86 machines (but uses -mcpu=i686 optimization for best performance on i686-class machines like the P3, P4, and Duron/Athlon).
http://www.slackware.com/info/

So, if you have a high-end pentium and you want it to run on all i686 processors you can do:

Code:

-march=i686 -mtune=YOUR_PROCESSOR_GOES HERE
instead of just

Code:

-march=YOUR_PROCESSOR_GOES_HERE

GrapefruiTgirl 06-07-2007 01:17 PM

grrrrrrrrrrrrrrr.. gee cee cee shmeee cee cee
 
okies.. Well, glibc compiled and tested out perfectly, a score of 100% :) and following that, GCC 4.1.2 also built, and this time, passed all tests with flying colors.
Also, I have the package-making part figured out now for sure.

However, after building GCC, the next thing is binutils. The ./configure instruction for it craps out, telling me "gcc cannot compile executables."

So, prior to making GCC, the sanity checks were all good.
ldd --version returns "glibc 2.4"
gcc --version returns "gcc 4.1.2"

so this is fine. However, here is the docs for installing gcc:
Code:

make -k check || :
../gcc-${GCC_VER}/contrib/test_summary

make DESTDIR=$PM_DEST install -j1

mkdir -pv $PM_DEST/lib
ln -sfv ../usr/bin/cpp $PM_DEST/lib
ln -sfv gcc $PM_DEST/usr/bin/cc

And something there doesn't make sence to me. Here's what I did (with MY $PM_DEST) :
Code:

make -k check || :
../gcc-${GCC_VER}/contrib/test_summary

make DESTDIR=/tmp/gcc412 install -j1

mkdir -pv /tmp/gcc412/lib
ln -sfv ../usr/bin/cpp /tmp/gcc412/lib
ln -sfv gcc /tmp/gcc412/usr/bin/cc

Now, this doesn't seem to make sense; I mean, there's at the very least, a missing 'cd..' or two in these instructions; but ultimately, why the ^$%#% am I sym-linking 'gcc' to the $PM_DEST location?
I tried making & installing this package 4 or 5 times now, both in BETWEEN the MAKE and MKDIR commands, followed by symlinking, and also AFTER doing the make & mkdir & symlinks commands, and still, when I run ./configure, it finds gcc ok, but then stops and tells me gcc cannot make executables.
Something's obviously not where it ought to be...
Anyway, I have saved ALL 5 GCC packages I have made, so it isn't hard for me to re-package it, if I can figure out what the heck's going on here and correct it.

{ example of making my packages, for the record:
# cd /tmp/gcc412
# makepkg -l y -c n /tmp/gcc-4.1.2-i686-6.tgz
# cd ..
# installpkg gcc-4.1.2-i686-6.tgz }

@ H_TexMeX_H -- :) Looks like I'm basically doing an upgrade and 'some' optimization when compared to the original Slackware CFLAGS (I have a P4) but the sense of accomplishment will be well worth it when I get there :D

:confused:-------------------->:confused::confused:-------------------------------------->:confused::confused::confused::confused:

EDIT: I'm making the compiler over again.


All times are GMT -5. The time now is 07:13 AM.