[SOLVED] Asking review of slackpkg script . @bassmadrigal & chrisretusn

babydr · 04-21-2021, 08:40 PM

Hello Bassmadrigal & Chrisretusn ,

Requesting review of the script below for when 15.+ becomes published .
Looking for suggestions of better methodologies &/or readability .

Tia , JimL

Code:

#!/bin/bash
# Note: this shell script uses slackpkg-15.X and newer functions .
#
# V2a(202104211510) ,  Jim Laferriere <babydr@baby-dragons.com>
# V1c(201810181209) ,  "
#

# Tell us what version of slackware/slamd64 we are running
#
echo -en "\n$(cat /etc/slackware-version)\n\n"
cat << PROGDB
#
# Updating 'The package list' ie: program database .
#

PROGDB

slackpkg -batch=on -default_answer=y update


aAa=$(ls /var/lib/pkgtools/packages/slackpkg* | awk -F'/' '{print $6}')
bBb=$(slackpkg info slackpkg | grep "PACKAGE[[:space:]]NAME:" | awk -F":" '{print $2}' | sed -e's/[[:space:]]//g' -e's/\.txz$//')

if [ "${aAa}" != "${bBb}" ]; then

  cat << PROGSLACKPKG
#
# Upgrading from ${aAa} to ${bBb}
#
# will also reload 'The package list' as successful install 
# will remove the files needed for the remainder of the script .
#

PROGSLACKPKG

  slackpkg -batch=on -default_answer=y upgrade slackpkg

  slackpkg -batch=on -default_answer=y update

fi


cat << PROGNEW
#
# Install Any NEW packages .
#

PROGNEW

slackpkg -batch=on -default_answer=y install-new


cat << PROGUPG
#
# Upgrade installed packages that are updated .
#

PROGUPG

slackpkg -batch=on -default_answer=y upgrade-all

exit

lovemeslk · 04-22-2021, 01:24 PM

cute script why ? Sure it works for you Why.
https://mirrors.kernel.org/slackware...nt/UPGRADE.TXT

Code:

   #!/bin/sh
    for dir in a ap d e f k kde l n t tcl x xap xfce y ; do
      ( cd $dir ; upgradepkg --install-new *.t?z )
    done

I really think slackpg is th stuff though.
when it updates your sytem and your sbo builds
and your graphics drivers and 32 bit like mine does let me know.
That is a pretty script I guess by looking should upgrade your system.
Yeap it works.

igadoter · 04-22-2021, 01:55 PM

I think there will be no one enough ... courageous to try your script - so how to review?

shruggy · 04-22-2021, 02:00 PM

Quote:

Originally Posted by babydr

Code:

aAa=$(ls /var/lib/pkgtools/packages/slackpkg* | awk -F'/' '{print $6}')
bBb=$(slackpkg info slackpkg | grep "PACKAGE[[:space:]]NAME:" | awk -F":" '{print $2}' | sed -e's/[[:space:]]//g' -e's/\.txz$//')

Nitpicking a bit. I don't like the code above. My take on it

Code:

aAa=$(basename /var/lib/pkgtools/packages/slackpkg*)
bBb=$(slackpkg info slackpkg|awk -F': *|\.txz' '/PACKAGE NAME:/,$0=$2')

chrisretusn · 04-23-2021, 09:32 AM

When I get a chance I will give the script a try.

pan64 · 04-23-2021, 10:16 AM

useless use of cat

bassmadrigal · 04-23-2021, 11:08 AM

Quote:

Originally Posted by babydr

Hello Bassmadrigal & Chrisretusn ,

Requesting review of the script below for when 15.+ becomes published .
Looking for suggestions of better methodologies &/or readability .

I'd probably define a slackpkg options variable up top (maybe SLKPKG_OPT). That way if slackpkg changes or you want to adjust the options, you can simply adjust a single line up top rather than every command in the script.

Code:

SLKPKG_OPT="-batch=on -default_answer=y"
slackpkg $SLKPKG_OPT update

Also, from an efficiency standpoint, using slackpkg info takes more processing time than interacting directly with the ChangeLog.txt. Although, this is just nitpicky since we're talking fractions of a second and only a single command in this instance. However, when scripting/programming, these small inefficiencies in a single command can cause exponential increases in time depending on what's being done.

In other words, it doesn't matter with something small like this, but in a more complex program, it could slow down things significantly.

Code:

jbhansen@craven-moorhead:~$ time slackpkg info slackpkg | grep "PACKAGE[[:space:]]NAME:" | awk -F":" '{print $2}' | sed -e's/[[:space:]]//g' -e's/\.txz$//'
slackpkg-2.82.1-noarch-3

real    0m0.071s
user    0m0.060s
sys     0m0.027s
jbhansen@craven-moorhead:~$ time grep ^ap/slackpkg /var/lib/slackpkg/ChangeLog.txt | head -n1 | cut -d: -f1 | cut -d/ -f2
slackpkg-2.82.1-noarch-3.txz

real    0m0.004s
user    0m0.003s
sys     0m0.004s

.004 vs .071 seconds in this instance is nothing. But if you had a script that had to loop over this a hundred times, yours would take 7.1 seconds where mine would be .4 seconds.

Quote:

Originally Posted by igadoter

I think there will be no one enough ... courageous to try your script - so how to review?

It's a straight forward script. If you look at the slackpkg lines, it's easy to see what's happening:

Code:

slackpkg -batch=on -default_answer=y update
slackpkg -batch=on -default_answer=y upgrade slackpkg
slackpkg -batch=on -default_answer=y update
slackpkg -batch=on -default_answer=y install-new
slackpkg -batch=on -default_answer=y upgrade-all

The if statement is just checking whether there's a newer slackpkg on the server and will update that separately and then run the update again.

Quote:

Originally Posted by pan64

useless use of cat

Where? Unless you're referencing his output of multiple lines using cat and heredoc. I don't see that as a useless use of cat (UUOC).

A UUOC would be something like cat filename.txt | grep something, when you can simply do grep something filename.txt which gets the same result with less processing power. cat with heredoc is extremely common and in my limited searching online, is not considered UUOC by pretty much anyone.

pan64 · 04-23-2021, 11:33 AM

Quote:

Originally Posted by bassmadrigal

useless use of cat

Where? Unless you're referencing his output of multiple lines using cat and heredoc. I don't see that as a useless use of cat (UUOC).

A UUOC would be something like cat filename.txt | grep something, when you can simply do grep something filename.txt which gets the same result with less processing power. cat with heredoc is extremely common and in my limited searching online, is not considered UUOC by pretty much anyone.

Strictly UUOC means any invocation of cat where it is just an overhead. What you explained is only the most frequent (obvious) appearance, but there are other cases.

Here are some other cases:
echo "$(cat file)"
this is most probably useless use of echo, it is just a simple cat file

And what about your heredoc?

Code:

cat << PROGDB
#
# Updating 'The package list' ie: program database .
#

PROGDB

is most probably identical to:

Code:

echo "
#
# Updating 'The package list' ie: program database .
#

"

additionally there is a useless use of pipe:
as you stated

Code:

slackpkg info slackpkg | grep "PACKAGE[[:space:]]NAME:" | awk -F":" '{print $2}' | sed -e's/[[:space:]]//g' -e's/\.txz$//'

is inefficient and have a better way, like:

Code:

grep ^ap/slackpkg /var/lib/slackpkg/ChangeLog.txt | head -n1 | cut -d: -f1 | cut -d/ -f2

which is definitely better, but still UUOP, because it can be solved without any pipe, with a single awk (for example).

From my point of view all of them (UUOCat, UUOEcho, UUOPipe and friends) are just wasting resources. I think in gereral UUOC covers all of these and similar cases.

shruggy · 04-23-2021, 11:38 AM

Quote:

Originally Posted by bassmadrigal

Code:

grep ^ap/slackpkg /var/lib/slackpkg/ChangeLog.txt | head -n1 | cut -d: -f1 | cut -d/ -f2

I wonder how awk would compare here?

Code:

awk -F'/|\.txz:' '/^ap\/slackpkg/{print$2;exit}' /var/lib/slackpkg/ChangeLog.txt

bassmadrigal · 04-23-2021, 12:34 PM

Quote:

Originally Posted by pan64

And what about your heredoc?

Code:

cat << PROGDB
#
# Updating 'The package list' ie: program database .
#

PROGDB

is most probably identical to:

Code:

echo "
#
# Updating 'The package list' ie: program database .
#

"

I don't think this type of UUOC is what was intended by the UUOC award. This is used widely in shell scripting and barely has any noticeable difference in overhead. I typically prefer using echo, but I wouldn't be calling someone on a UUOC for something simple like this and I don't think the creators of the UUOC award would either.

Quote:

Originally Posted by pan64

additionally there is a useless use of pipe:
as you stated

Code:

slackpkg info slackpkg | grep "PACKAGE[[:space:]]NAME:" | awk -F":" '{print $2}' | sed -e's/[[:space:]]//g' -e's/\.txz$//'

is inefficient and have a better way, like:

Code:

grep ^ap/slackpkg /var/lib/slackpkg/ChangeLog.txt | head -n1 | cut -d: -f1 | cut -d/ -f2

which is definitely better, but still UUOP, because it can be solved without any pipe, with a single awk (for example).

Yes, you can do it without a pipe, but that isn't always better. In this instance, the awk command provided below by shruggy is about 40% slower to complete as piping over 1000 iterations.

Code:

jbhansen@craven-moorhead:~$ time for i in {1..1000}; do awk -F'/|\.txz:' '/^ap\/slackpkg/{print$2;exit}' /var/lib/slackpkg/ChangeLog.txt; done

real    0m4.878s
user    0m3.976s
sys     0m0.893s
jbhansen@craven-moorhead:~$ time for i in {1..1000}; do grep ^ap/slackpkg /var/lib/slackpkg/ChangeLog.txt | head -n1 | cut -d: -f1 | cut -d/ -f2; done

real    0m2.976s
user    0m3.108s
sys     0m2.521s

Quote:

Originally Posted by shruggy

I wonder how awk would compare here?

Code:

awk -F'/|\.txz:' '/^ap\/slackpkg/{print$2;exit}' /var/lib/slackpkg/ChangeLog.txt

On a single run, awk was only slightly slower, but it did generate a warning on my 14.2 system:

Code:

jbhansen@craven-moorhead:~$ time awk -F'/|\.txz:' '/^ap\/slackpkg/{print$2;exit}' /var/lib/slackpkg/ChangeLog.txt
awk: warning: escape sequence `\.' treated as plain `.'
slackpkg-2.82.1-noarch-3

real    0m0.006s
user    0m0.005s
sys     0m0.001s

However, as when seen above, when looped over 1000 iterations, it is almost twice as slow as piping.

And just for kicks, here's how long the slackpkg info command takes when looping it 1000 times:

Code:

time for i in {1..1000}; do slackpkg info slackpkg | grep "PACKAGE[[:space:]]NAME:" | awk -F":" '{print $2}' | sed -e's/[[:space:]]//g' -e's/\.txz$//'; done

real    1m14.980s
user    1m1.178s
sys     0m28.105s

shruggy · 04-23-2021, 12:58 PM

Quote:

Originally Posted by bassmadrigal

it did generate a warning on my 14.2 system

Ah this is gawk, so the backslash must be escaped: awk -F'/|\\.txz:'.

Actually, gawk is rather sluggish compared to mawk and BWK awk (The One True Awk).

pan64 · 04-23-2021, 01:08 PM

Quote:

Originally Posted by bassmadrigal

I don't think this type of UUOC is what was intended by the UUOC award. This is used widely in shell scripting and barely has any noticeable difference in overhead. I typically prefer using echo, but I wouldn't be calling someone on a UUOC for something simple like this and I don't think the creators of the UUOC award would either.

That is still UUOC, using external program instead of builtin is just huge waste of resources. You can measure it yourself.

Quote:

Originally Posted by bassmadrigal

Yes, you can do it without a pipe, but that isn't always better. In this instance, the awk command provided below by shruggy is about 40% slower to complete as piping over 1000 iterations.

Code:

jbhansen@craven-moorhead:~$ time for i in {1..1000}; do awk -F'/|\.txz:' '/^ap\/slackpkg/{print$2;exit}' /var/lib/slackpkg/ChangeLog.txt; done

real    0m4.878s
user    0m3.976s
sys     0m0.893s
jbhansen@craven-moorhead:~$ time for i in {1..1000}; do grep ^ap/slackpkg /var/lib/slackpkg/ChangeLog.txt | head -n1 | cut -d: -f1 | cut -d/ -f2; done

real    0m2.976s
user    0m3.108s
sys     0m2.521s

Obviously not always better. It depends on the implementation. I guess the complex field separator is the issue now, but otherwise every pipe will cause an additional socket and fork (both costs a lot). So probably there can be a more efficient implementation in awk....

bassmadrigal · 04-23-2021, 01:55 PM

Quote:

Originally Posted by pan64

That is still UUOC, using external program instead of builtin is just huge waste of resources. You can measure it yourself.

I did measure it. It was 0.000s for echo and 0.001s for cat. So "huge waste of resources" is extreme exaggeration when using cat << EOF. The memory usage difference would probably be laughably small as well.

Also, cat << EOF can be very beneficial when using variables or quotes. Using echo might be slightly faster, but it would be at the expense of complicating things for the programmer.

Quote:

Originally Posted by pan64

Obviously not always better. It depends on the implementation. I guess the complex field separator is the issue now, but otherwise every pipe will cause an additional socket and fork (both costs a lot). So probably there can be a more efficient implementation in awk....

But is a potential slight savings worth the complexity in the awk command? As well as the extra time in figuring out how to make your awk command more efficient?

Ultimately, for UUO*, is it better to save computer's resources (CPU time and RAM usage) or the programmer's resources (time and mental effort)? Sometimes both, but that certainly isn't always the case. Obviously, it will depend on the implementation. Some may argue it's always better to save computer resources, but that's the opinion of that person and not a fact that everyone would agree on.

In this script, cat << EOF could go either way, in other scripts with quotes or variables, cat << EOF would probably be worth the penalty in computer resources to save the programmer's time in needing to escape all the various things that might need to be escaped. In this script, the pipes would probably be preferably to the amount of time it might take to improve awk's resource usage, especially for those not very familiar with awk.

Are there times where UUO* is a waste that (almost) everyone can agree on? Absolutely! But UUO* should not be a bible that scripters/programmers should live and die by. Learn about it and find ways to improve your scripting/programming by using it when it makes sense.

pan64 · 04-23-2021, 03:10 PM

Quote:

Originally Posted by bassmadrigal

I did measure it. It was 0.000s for echo and 0.001s for cat. So "huge waste of resources" is extreme exaggeration when using cat << EOF. The memory usage difference would probably be laughably small as well.

Obviously it is not a usable measurement. 0 is not comparable. (you can say 0.001s is more than million/billion times larger). You need to be more precise. But it is quite difficult.
First, there are other processes running, so the result depends on the load. Next, the binaries (the code of the executables) are cached, so you need to take it into account (if grep/awk/cat/bash is actually used from RAM or have to be loaded). Additionally you need to find something measurable, as you see the time itself is pointess/useless.
Believe me, a simple echo is 1000 times faster than a cat heredoc (if both bash and cat are cached, so you need to measure only the fork/clone). And we need to measure not only the time, but probably memory usage and others.
And you are right, it is negligible if you are all alone on an idle host.

Quote:

Originally Posted by bassmadrigal

Also, cat << EOF can be very beneficial when using variables or quotes. Using echo might be slightly faster, but it would be at the expense of complicating things for the programmer.

Obviously there are cases when heredoc and/or cat is really useful. But it is not the initial post.

Quote:

Originally Posted by bassmadrigal

But is a potential slight savings worth the complexity in the awk command? As well as the extra time in figuring out how to make your awk command more efficient?

It is not slight savings, as it was explained, fork and pipes are expensive.
By default we would need to compare the same pattern using grep/cut/head/whatever pipe-chains and a single awk. Again, using the same pattern. There is no extra effort to make it better, just use the built-in features of awk instead of piping every logical step into a different tool. And you will see again the extra cost of additional forks. Not to speak about the fact that [almost] all of these tools will do a loop on the lines of the input file. How many loops do you really need (I guess only one).

Quote:

Originally Posted by bassmadrigal

Ultimately, for UUO*, is it better to save computer's resources (CPU time and RAM usage) or the programmer's resources (time and mental effort)? Sometimes both, but that certainly isn't always the case. Obviously, it will depend on the implementation. Some may argue it's always better to save computer resources, but that's the opinion of that person and not a fact that everyone would agree on.

It is about the usage and usability. If you want to execute it only once you can do whatever you want. But if you need to execute it repeatedly you [would] want to make it in an efficient way.

Quote:

Originally Posted by bassmadrigal

Are there times where UUO* is a waste that (almost) everyone can agree on? Absolutely! But UUO* should not be a bible that scripters/programmers should live and die by. Learn about it and find ways to improve your scripting/programming by using it when it makes sense.

I give you an example: if you want to go from A to B you will always want to find the optimal way. Depends on your capabilities it can be the fastest, shortest way. This is how we write programs. Do you want a fast, reliable, lightweight, robust solution?
But anyway, what is your expectation if you see a new script for 15+?

igadoter · 04-23-2021, 03:36 PM

Quote:

Originally Posted by pan64

It is not slight savings, as it was explained, fork and pipes are expensive.

This reminds me a joke about Italian in London restaurant - person has very bad English pronunciation - just before being thrown out into the street - he said "I want sheet on my table".