SlackwareThis Forum is for the discussion of Slackware Linux.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
@Woodsman I hope you did consider that if a SlackBuild is stable and builds the way it is for a specific release, changing the CPU type flag (march/mtune) for it is unlikely to give troubling difference when rebuilding. Unless the binary could really be only built manually. I find that unlikely, but again we could make use of a customized SlackBuild.
Sure, I get that. My point is, who has the time? The person who wants a highly customized operating system is the exception and not the norm. Way, way out there at the end of the bell curve. Distro maintainers do not make distros for the exception, they make them for the norm. Basic economics, basic return on investment, best bang for the buck.
Ok I guess I won't be able to prove that it's possible unless I check it myself. I'll try to see if a script could be made that would adopt to every type of SlackBuild that's there. Yet again if it's really wanted the SlackBuilds could simply just be revised to make them adoptable to cpu-type-specific build and it's only done once. We could even create a script that would change multiple SlackBuilds if their signatures are similar.
And you're not adding a major architecture. The targets would still be x86_64 and ix86.
This is TOTALLY irrelevant and shows that you have either not read my posts at all or do not understand the words I am typing. NOT ALL PACKAGES WILL COMPILE, EVEN FOR THE SAME ARCHITECTURE WITH THE SAME OPTIMIZATIONS THAT WERE USED TO BUILD THEM ORIGINALLY. Read that and re-read it until you understand it. If you still don't understand it, then you are not at a sufficient level to discuss this further.
Quote:
Originally Posted by konsolebox
@ruario Once I had attempted to use packages of Slackware for Gentoo and it worked. Yes they could be identical, at least signifcantly.
Gentoo and Slackware have architectural differences at their very core. Gentoo builds exactly what you tell it to, while Slackware ships the kitchen sink and links against almost everything it can. Comparing the two distros will tell you very little about whether or not the optimizations were the causal factor in any perceived difference.
[edit]
Quote:
Originally Posted by konsolebox
Ok I guess I won't be able to prove that it's possible unless I check it myself. I'll try to see if a script could be made that would adopt to every type of SlackBuild that's there. Yet again if it's really wanted the SlackBuilds could simply just be revised to make them adoptable to cpu-type-specific build and it's only done once. We could even create a script that would change multiple SlackBuilds if their signatures are similar.
Again, see Alien Bob's ARM port. He has already started modifying some SlackBuilds to support centralized ARCH and SLKCFLAGS values. This may or may not find its way into official Slackware, but if I had to choose a starting point, that would be it.
This is TOTALLY irrelevant and shows that you have either not read my posts at all or do not understand the words I am typing. NOT ALL PACKAGES WILL COMPILE, EVEN FOR THE SAME ARCHITECTURE WITH THE SAME OPTIMIZATIONS THAT WERE USED TO BUILD THEM ORIGINALLY. Read that and re-read it until you understand it. If you still don't understand it, then you are not at a sufficient level to discuss this further.
Oh sorry. Being able to compile packages in Gentoo and even vanilla packages with cpu-specific optimizations didn't give trouble so far so I have my doubts about that.
And perhaps there would be exemptions alright - with old packages probably, or perhaps on older gcc versions. But if that is the case you could just pick which packages could have their unoptimized versions. It's rare for one package to not build just by changing the target cpu type. And again about that gcc thing, I don't think it would be applicable since we won't be using other gcc's anyway but the one in that specific release.
Quote:
Gentoo and Slackware have architectural differences at their very core. Gentoo builds exactly what you tell it to, while Slackware ships the kitchen sink and links against almost everything it can. Comparing the two distros will tell you very little about whether or not the optimizations were the causal factor in any perceived difference.
I'm not sure how an executable being able to link up to many libraries and having more code affect the hot spots of it. An increase in memory perhaps.
Quote:
Again, see Alien Bob's ARM port. He has already started modifying some SlackBuilds to support centralized ARCH and SLKCFLAGS values. This may or may not find its way into official Slackware, but if I had to choose a starting point, that would be it.
Thanks but I already studied some of the SlackBuilds. I think one signature block could be modified to allow accepting an exported variable like CPU in which decisions about which CFLAGS to be used could be based upon. This would apply if the SlackBuild has it. If it doesn't apply to the SlackBuild I guess a custom SlackBuild should be made for it. I'm estimating at least 90% of packages would apply to it but I haven't tried it yet. Perhaps I'll be doing the scripting and assessment later.
The modified version could be used on the fly through a pipe or process substition or stored as another file.
Last edited by konsolebox; 04-18-2013 at 12:21 AM.
@konsolebox: reading your answers, I think that the only way you can begin to convince others is exhibiting at least a small but working example of what you magical scripts could look like, successfully applied to a significant set of packages. Only then that discussion can become practical, not theoretical. Theory is only interesting if you can probe it.
konsolebox, honestly, it seems to me that you make a lot of assumptions and take those for granted but you really have no idea of the work needed to do what you want to do and the maintenability of it.
Quote:
Oh sorry. Being able to compile packages in Gentoo and even vanilla packages with cpu-specific optimizations didn't give trouble so far so I have my doubts about that.
ebuilds and slackbuilds are very different: in Gentoo (I used it for some time) one day, if you're very lucky, you can build everything you need on a specific ARCH and with specific CFLAGS, the next day not, and they have lots of maintainers. On Funtoo (I used it for some time) the situation is worse as they have less maintainers. On Slackware there's just one maintainer.
upgrading an already installed Gentoo system very rarely works, you have to start from scratch: can you think how is painful and time/resources consuming is to rebuild everything from scratch when you have to upgrade some components (assumed all goes well, and usually it's not)? Then you should also test everything you have rebuilt on every platform, as you can't offer untested packages and guarantee they will work.
that's why I suggested you to try, to verify your assumptions: as long as you guess only you can't have an idea on how you are wrong.
and when you eventually are successful in doing it (but I strongly doubt that you will be) in multiple archs, upgrade glibc and gcc (just to name 2 software) and try again.
I suppose by then you will understand.
Yet again if it's really wanted the SlackBuilds could simply just be revised to make them adoptable to cpu-type-specific build and it's only done once. We could even create a script that would change multiple SlackBuilds if their signatures are similar.
My old doctor in my native Austria had a nice saying for that:
If my grandmother had wheels, she would be a bus, and I could go for a ride with her.
Or as Linus Torvalds himself stated in another context, namely the SCO lawsuit:
Whether you're doing it for your own machine or for your target population, who can stop you? Think of it this way, build it for your own machine, if you like it then it's all good and you can stick with it. If you feel like sharing, then you can come back here for some feedback or perhaps ask politely if someone wants to try your build.
But then again I think someone mentioned before that computers these days are already fast enough. You have to remember that in the end computer is just a tool. No matter how fast your computer is, things will still depend on your attitude. I bet the time saved from your fast processing power is already used up for arguing too much here (but healthy argument is fine)
Talk less, do more, isn't that Slackware community's culture?
So this time I finally had enough time to try materializing the concept. And so I was able to create this script that would try to walk on all SlackBuilds available in the source directory.
Code:
#!/bin/bash
#TEMPDIR=/tmp
DESTDIR=.
DRY_RUN=false
OVERWRITE_MODE=false
VERBOSE_MODE=false
[[ BASH_VERSINFO -ge 3 ]] || {
echo "You need Bash version 3.0 or newer to run this script."
exit 1
}
shopt -s extglob
function getabspath {
local -a T1 T2
local -i I=0
local IFS=/ A
case "$1" in
/*)
read -r -a T1 <<< "$1"
;;
*)
read -r -a T1 <<< "/$PWD/$1"
;;
esac
T2=()
for A in "${T1[@]}"; do
case "$A" in
..)
[[ I -ne 0 ]] && unset T2\[--I\]
continue
;;
.|'')
continue
;;
esac
T2[I++]=$A
done
case "$1" in
*/)
[[ I -ne 0 ]] && __="/${T2[*]}/" || __=/
;;
*)
[[ I -ne 0 ]] && __="/${T2[*]}" || __=/.
;;
esac
}
function verbose {
[[ $VERBOSE_MODE == true ]] && echo "$1"
}
function save_lines {
local IFS=$'\n'
echo "${LINES[*]}" > "$1"
}
function parse_args {
while [[ $# -gt 0 ]]; do
case "$1" in
-d|--dest-dir)
if [[ -z $2 ]]; then
echo "Option $1 needs an argument."
return 1
fi
DEST_DIR=$2
shift
;;
--dry-run)
DRY_RUN=true
;;
-o|--overwrite)
OVERWRITE_MODE=true
;;
-v|--verbose)
VERBOSE_MODE=true
esac
shift
done
}
function main {
getabspath "$PWD/"
CURRENTDIRNODE=$__
getabspath "$DESTDIR/"
DESTDIRPREFIX=$__
if [[ $DESTDIRPREFIX == "$CURRENTDIRNODE" ]]; then
DESTDIRPREFIX=''
elif [[ $DRY_RUN = false && ! -d $DESTDIR ]]; then
mkdir -p "$DESTDIR" || {
echo "Unable to create destination directory: $DESTDIR"
return 1
}
fi
# if [[ ! -r $TEMPDIR || ! -w $TEMPDIR || ! -x $TEMPDIR ]]; then
# echo "Temporary directory is not accessible or writeable: $TEMPDIR"
# return 1
# fi
# getabspath "$TEMPDIR/"
# if [[ $__ == "$CURRENTDIRNODE" ]]; then
# TEMPDIRPREFIX=''
# else
# TEMPDIRPREFIX=$__
# fi
local DIRBASE FILE LINE IFS=''
local FILES_COUNTER=0
while read -u 3 FILE; do
echo "Processing file $FILE."
DIRBASE=${FILE%%/+([^/])}
FILEBASE=${FILE##*/}
FINALDESTDIR=${DESTDIRPREFIX}${DIRBASE}
if [[ $DRY_RUN == false && ! -d $FINALDESTDIR ]]; then
mkdir -p "$FINALDESTDIR" || {
echo "Unable to create destination directory: $FINALDESTDIR"
return 1
}
fi
TARGET=${DESTDIRPREFIX}${FILE}.custom
if [[ $OVERWRITE_MODE == false && -e $TARGET ]]; then
echo "Skipping file $FILE: $TARGET already exists."
FILES_SKIPPED_EXISTING[FILES_COUNTER++]=$FILE
continue
fi
# TEMPFILE=${TEMPDIRPREFIX}${FILEBASE}.custom.temp
# : > "$TEMPFILE" || {
# echo "Unable to create or truncate temporary file $TEMPFILE."
# return 1
# }
CONVERT_SLKCFLAGS_COUNT=0
CONVERT_MAKEPKG_COUNT=0
CONVERT_CFLAGS=0
LINES=()
LINES_COUNT=0
while read -u 4 LINE; do
case "$LINE" in
*([[:blank:]])'elif [ "'@('$ARCH'|'${ARCH}')'" = "x86_64" ]; then'*([[:blank:]]))
verbose "Detected $LINE."
LINES[LINES_COUNT++]=$LINE
read -u 4 LINE || break
case "$LINE" in
*([[:blank:]])'SLKCFLAGS="-O'?([[:digit:]])' -fPIC"'*([[:blank:]]))
verbose "Detected $LINE."
LINES[LINES_COUNT++]=${LINE/'-O'/'-march=core2 -O'}
(( ++CONVERT_SLKCFLAGS_COUNT ))
;;
*)
LINES[LINES_COUNT++]=$LINE
;;
esac
;;
*([[:blank:]])?(/sbin/)makepkg\ *@('$ARCH'|'${ARCH}')-@('$BUILD'|'${BUILD}').@(txz|tgz|tbz2)*([[:blank:]]))
verbose "Detected $LINE."
LINES[LINES_COUNT++]=${LINE/@('$ARCH'|'${ARCH}')-@('$BUILD'|'${BUILD}')/'${ARCH}_core2-${BUILD}'}
(( ++CONVERT_MAKEPKG_COUNT ))
;;
*CFLAGS*|*CXXFLAGS*)
verbose "Detected $LINE."
(( ++CONVERT_CFLAGS ))
LINES[LINES_COUNT++]=$LINE
;;
*)
LINES[LINES_COUNT++]=$LINE
;;
esac
done 4< "$FILE"
if (( CONVERT_SLKCFLAGS_COUNT == 1 && CONVERT_MAKEPKG_COUNT == 1 )); then
if [[ $DRY_RUN = true ]] || save_lines "$TARGET"; then
echo "Converted file $FILE as $TARGET."
FILES_CONVERTED[FILES_COUNTER++]=$FILE
else
echo "Unable to create file or write data to file $TARGET."
FILES_UNCONVERTED_GENERAL_FAILURE[FILES_COUNTER++]=$FILE
fi
elif (( CONVERT_SLKCFLAGS_COUNT == 0 && CONVERT_CFLAGS == 0 && CONVERT_MAKEPKG_COUNT >= 1 )); then
echo "Not converting file $FILE: Probably doesn't compile or is meant to be not optimized."
FILES_UNCONVERTED_NO_COMPILE[FILES_COUNTER++]=$FILE
elif (( (CONVERT_SLKCFLAGS_COUNT >= 1 || CONVERT_CFLAGS >=1 ) && CONVERT_MAKEPKG_COUNT == 0 )); then
echo "Not converting $FILE: Probably compiles its packages but needs to be converted manually."
FILES_UNCONVERTED_COMPILES[FILES_COUNTER++]=$FILE
else
echo "Not converting file $FILE: Type is generally unrecognized and needs to be checked and/or converted manually."
FILES_UNCONVERTED_UNSUAL_TYPE[FILES_COUNTER++]=$FILE
fi
echo
done 3< <(exec find -maxdepth 3 -mindepth 3 -type f -readable -iname '*.SlackBuild' | sort -t / -k 2,2)
IFS=$'\n'
echo
echo "# Files Converted"
echo
if [[ ${#FILES_CONVERTED[@]} -gt 0 ]]; then
echo "${FILES_CONVERTED[*]}"
echo
fi
echo "# Files not converted in which packages are probably not compiled or is meant to be not optimized."
echo
if [[ ${#FILES_UNCONVERTED_NO_COMPILE[@]} -gt 0 ]]; then
echo "${FILES_UNCONVERTED_NO_COMPILE[*]}"
echo
fi
echo "# Files not converted in which packages are probably compiled but needs to be converted manually."
echo
if [[ ${#FILES_UNCONVERTED_COMPILES[@]} -gt 0 ]]; then
echo "${FILES_UNCONVERTED_COMPILES[*]}"
echo
fi
echo "# File not converted due to general failure."
echo
if [[ ${#FILES_UNCONVERTED_GENERAL_FAILURE[@]} -gt 0 ]]; then
echo "${FILES_UNCONVERTED_GENERAL_FAILURE[*]}"
echo
fi
echo "# Files skipped as they already exists."
echo
if [[ ${#FILES_SKIPPED_EXISTING[@]} -gt 0 ]]; then
echo "${FILES_SKIPPED_EXISTING[*]}"
echo
fi
}
parse_args "$@" && main "$@"
So the purpose of the script is to create custom SlackBuild files along the same directories or on a custom directory.
And on my last attempt to run the script this is what I got:
So as a summary we have 638 SlackBuilds processed.
488 or 76.48% of those were converted automatically.
139 or 21.78% of those were not converted since files in the package it represents are probably not compiled, or are compilable but are meant to be not optimized, or were just chosen to be not optimized, and probably some other reasons as well.
And 11 of those or 1.72% has an unusual format and needs to be checked manually.
So that's were things have gone so far. I thought to make an update about it.
The next step is to create another script that would finally build the packages that could be rebuilt, then build it. The last would be putting everything together and making the ISO. I won't rush it and always take my easy time on it.
You might have noticed that the changes I made were basically only two lines, but it would improved when I finally decide to think about which CPUs are worth adding to those. But I think the concept of automatic conversion of the scripts is proven already.
konsolebox, do you know that slackware is maintained by just one person?
I think you are missing this very important detail related to what you're asking for (and that is the reason of my previous post).
yes Pat works hard and Ponce your work I have used so many times you also are top shelf. Thank you for all your wonderful Slack.
Quote:
So as a summary we have 638 SlackBuilds processed.
488 or 76.48% of those were converted automatically.
139 or 21.78% of those were not converted since files in the package it represents are probably not compiled, or are compilable but are meant to be not optimized, or were just chosen to be not optimized, and probably some other reasons as well.
And 11 of those or 1.72% has an unusual format and needs to be checked manually.
So that's were things have gone so far. I thought to make an update about it.
The next step is to create another script that would finally build the packages that could be rebuilt, then build it. The last would be putting everything together and making the ISO. I won't rush it and always take my easy time on it.
You might have noticed that the changes I made were basically only two lines, but it would improved when I finally decide to think about which CPUs are worth adding to those. But I think the concept of automatic conversion of the scripts is proven already.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.