Boost "slackpkg upgrade-all" performance

grissiom · 08-21-2009, 07:09 PM

Talk is cheap, patch here: (against slackpkg-2.80.2-noarch-3)

Code:

--- /ext4/slackware_rsync/slackware64-current/slackware64/ap/slackpkg-2.80.2-noarch-3/usr/libexec/slackpkg/core-functions.sh  2009-07-27 09:33:49.000000000 +0800
+++ /usr/libexec/slackpkg/core-functions.sh     2009-08-22 07:41:30.106445523 +0800
@@ -567,7 +567,11 @@
                        done
                ;;
                upgrade-all)
-                       for i in $(cut -f2 -d\  ${TMPDIR}/tmplist); do
+                       cut -f2-6 -d\  ${TMPDIR}/tmplist > ${TMPDIR}/lpkg
+                       cut -f2-6 -d\  ${TMPDIR}/pkglist > ${TMPDIR}/spkg
+                       potential=$(cat ${TMPDIR}/{s,l}pkg | sort | uniq -u | cut -f1 -d\  - | uniq -d )
+                       #for i in $(cut -f2 -d\  ${TMPDIR}/tmplist); do
+                       for i in $potential; do

                                givepriority ${i}
                                [ ! "$FULLNAME" ] && continue

Before applying this patch, slackpkg will loop 800+ times which will take minutes for one "upgrade-all" on my box. After this patch, it only loop 5 times... There is even no time for spinning bar to appear ;D

Although I haven't tested it so much (I reverted two packages and "new slackpkg" can find them) but I think it would have no regressions. Any feedback is strongly welcome.

dive · 08-21-2009, 07:16 PM

You might want to send this to Piter Punk.

Bruce Hill · 08-21-2009, 08:11 PM

grissiom,

Are you updating from a local mirror, such as I do from my Slackware64 -current
mirror; or are you updating via the internut (Great Chinese Firewall version)?

I applied the patch and it took maybe 2 seconds for the line:

Code:

Looking for packages to upgrade. Please wait... DONE

to pop up in my terminal. Then the spinner appeared and BAM! the window with
the packages listed was there. And, yes, they are exactly the ones from the
ChangeLog that I had not updated, minus my blacklist such as kernel.

Thanks, and I too think you should send it to PiterPUNK. It's still his choice
what to do, and probably won't happen until Slackware-13.0 comes out.

grissiom · 08-21-2009, 10:13 PM

Quote:

Originally Posted by dive

You might want to send this to Piter Punk.

Yes, I have sent this to Piter. But I put it here to get more testing/feedback. Somebody wants to try something new may like it as well

grissiom · 08-21-2009, 10:22 PM

Quote:

Originally Posted by Bruce Hill

grissiom,

Are you updating from a local mirror, such as I do from my Slackware64 -current
mirror; or are you updating via the internut (Great Chinese Firewall version)?

Local mirror. But, anyway, rsync will have to go through the Great Firewall right?

Quote:

Originally Posted by Bruce Hill

I applied the patch and it took maybe 2 seconds for the line:

Code:

Looking for packages to upgrade. Please wait... DONE

to pop up in my terminal. Then the spinner appeared and BAM! the window with
the packages listed was there. And, yes, they are exactly the ones from the
ChangeLog that I had not updated, minus my blacklist such as kernel.

Glad to here that

Now I know the patch works on your box at least

Quote:

Originally Posted by Bruce Hill

Thanks, and I too think you should send it to PiterPUNK. It's still his choice
what to do, and probably won't happen until Slackware-13.0 comes out.

Yes, I mailed Piter. But it seems Piter is out for a vacation or something like that? I sent "search for .new file" patch to him weeks ago but don't get his reply yet...

niels.horn · 08-21-2009, 11:41 PM

Worked very well here!
Slackpkg was slow on my virtual machines especially, but after this patch it's a lot faster.

Thanks!

Bruce Hill · 08-22-2009, 12:03 AM

rsync doesn't have to go through the firewall, if you just use it local -- which I do
for everything where the files are important. Maybe you have better bandwidth in
Beijing that I do in a village here in Dali. Heck, the phone wires coming in to my
house to use the ADSL modem look like chicken intestines, so I doubt the copper
quality is very good. And they're hung all over roofs, trees, etc.

PiterPUNK most probably wouldn't submit any changes until after Slackware-13.0 anyway.

Pat surely has a freeze on anything that's not broken, and important.

Now you have niels.horn (also in PiterPUNK's area) using it with good results.

PiterPUNK really does appreciate us using and submitting suggestions for slackpkg.

Chuck56 · 08-22-2009, 07:52 AM

Quote:

Originally Posted by grissiom

Although I haven't tested it so much (I reverted two packages and "new slackpkg" can find them) but I think it would have no regressions. Any feedback is strongly welcome.

Feedback:

I am updating a desktop machine expecting to see the 8/20 thunderbird-2.0.0.23 patch from an LAN mirror via NFS. I replaced the 1 line (-) with the 5 lines (+) in /usr/libexec/slackpkg/core-functions.sh and got the following response.

Code:

root@slacker:~# slackpkg upgrade-all

Looking for packages to upgrade. Please wait... cut: /tmp/slackpkg.BK9mE8/pkglist: No such file or directory
DONE

No packages match the pattern for upgrade. Try:

        /usr/sbin/slackpkg install|reinstall


root@slacker:~#

When I backed out the changes it worked as expected. Did I do something wrong?

Bruce Hill · 08-22-2009, 08:00 AM

Please post the output of:

Code:

diff -u /usr/libexec/slackpkg/core-functions.sh /usr/libexec/slackpkg/core-functions.sh~

where /usr/libexec/slackpkg/core-functions.sh is the new file and /usr/libexec/slackpkg/core-functions.sh~ the old one.

Chuck56 · 08-22-2009, 08:10 AM

Quote:

Originally Posted by Chuck56

Did I do something wrong?

Self-inflicted wound...

I'm running slackpkg-2.70.5-noarch-1, not slackpkg-2.80.2-noarch-3. My modified feedback is that it doesn't appear to work in the stock 12.2 slackpkg.

grissiom · 08-22-2009, 10:35 AM

Quote:

Originally Posted by Bruce Hill

PiterPUNK most probably wouldn't submit any changes until after Slackware-13.0 anyway.

Pat surely has a freeze on anything that's not broken, and important.

Yes, it's reasonable.

Quote:

Now you have niels.horn (also in PiterPUNK's area) using it with good results.

PiterPUNK really does appreciate us using and submitting suggestions for slackpkg.

Yes, I think every developer's dream is getting people's suggestions and patches. Maybe he is busy this time. But I don't think he will miss it. Never mind~;-)

grissiom · 08-22-2009, 10:42 AM

Quote:

Originally Posted by Chuck56

Code:

root@slacker:~# slackpkg upgrade-all

Looking for packages to upgrade. Please wait... cut: /tmp/slackpkg.BK9mE8/pkglist: No such file or directory
DONE

No packages match the pattern for upgrade. Try:

        /usr/sbin/slackpkg install|reinstall


root@slacker:~#

When I backed out the changes it worked as expected. Did I do something wrong?

Thanks~ So I know the patch won't work with slackpkg-2.70.5-noarch-1. Make it to generate pkglist and apply my patch is just painful...

gegechris99 · 08-24-2009, 03:38 PM

Hello Grissiom,

I've used your patch and I'd like to report that "slackpkg upgrade-all" detected correctly the new upgraded packages for my -current machine.

And also, it was way much faster

Thank you.

BrZ · 08-30-2009, 08:17 AM

Thanks grissiom, worked fine on current 32 and 64.

gegechris99 · 09-14-2009, 02:12 PM

@ grissiom,

I'm sorry but it seems that your patch does not work for security updates to a stable release.

In my case, I used mirror: http://slackware.osuosl.org/slackware-13.0/.

Your patch failed to detect the 3 patches since release of 13.0 (namely, mozilla-firefox, seamonkey and seamonkey-solibs).

Could the reason be that file PACKAGES.TXT is not updated for a stable release (it's rather patches/PACKAGES.TXT) contrary to -current?