LinuxQuestions.org - Boost "slackpkg upgrade-all" performance

- Slackware (https://www.linuxquestions.org/questions/slackware-14/)

- - Boost "slackpkg upgrade-all" performance (https://www.linuxquestions.org/questions/slackware-14/boost-slackpkg-upgrade-all-performance-749412/)

grissiom

08-21-2009 07:09 PM

Boost "slackpkg upgrade-all" performance

Talk is cheap, patch here: (against slackpkg-2.80.2-noarch-3)

Code:

--- /ext4/slackware_rsync/slackware64-current/slackware64/ap/slackpkg-2.80.2-noarch-3/usr/libexec/slackpkg/core-functions.sh  2009-07-27 09:33:49.000000000 +0800

+++ /usr/libexec/slackpkg/core-functions.sh    2009-08-22 07:41:30.106445523 +0800

@@ -567,7 +567,11 @@

                        done

                ;;

                upgrade-all)

-                      for i in $(cut -f2 -d\  ${TMPDIR}/tmplist); do

+                      cut -f2-6 -d\  ${TMPDIR}/tmplist > ${TMPDIR}/lpkg

+                      cut -f2-6 -d\  ${TMPDIR}/pkglist > ${TMPDIR}/spkg

+                      potential=$(cat ${TMPDIR}/{s,l}pkg | sort | uniq -u | cut -f1 -d\  - | uniq -d )

+                      #for i in $(cut -f2 -d\  ${TMPDIR}/tmplist); do

+                      for i in $potential; do



                                givepriority ${i}

                                [ ! "$FULLNAME" ] && continue

Before applying this patch, slackpkg will loop 800+ times which will take minutes for one "upgrade-all" on my box. After this patch, it only loop 5 times... There is even no time for spinning bar to appear ;D

Although I haven't tested it so much (I reverted two packages and "new slackpkg" can find them) but I think it would have no regressions. Any feedback is strongly welcome.

dive	08-21-2009 07:16 PM

You might want to send this to Piter Punk.

Bruce Hill

08-21-2009 08:11 PM

grissiom,

Are you updating from a local mirror, such as I do from my Slackware64 -current
mirror; or are you updating via the internut (Great Chinese Firewall version)?

I applied the patch and it took maybe 2 seconds for the line:

Code:

Looking for packages to upgrade. Please wait... DONE

to pop up in my terminal. Then the spinner appeared and BAM! the window with
the packages listed was there. And, yes, they are exactly the ones from the
ChangeLog that I had not updated, minus my blacklist such as kernel.

Thanks, and I too think you should send it to PiterPUNK. It's still his choice
what to do, and probably won't happen until Slackware-13.0 comes out.

grissiom

08-21-2009 10:13 PM

Quote:

Originally Posted by dive (Post 3653064)

You might want to send this to Piter Punk.

Yes, I have sent this to Piter. But I put it here to get more testing/feedback. Somebody wants to try something new may like it as well ;)

grissiom

08-21-2009 10:22 PM

Quote:

Originally Posted by Bruce Hill (Post 3653095)

grissiom,

Are you updating from a local mirror, such as I do from my Slackware64 -current
mirror; or are you updating via the internut (Great Chinese Firewall version)?

Local mirror. But, anyway, rsync will have to go through the Great Firewall right? ;)

Quote:

Originally Posted by Bruce Hill (Post 3653095)

I applied the patch and it took maybe 2 seconds for the line:

Code:

Looking for packages to upgrade. Please wait... DONE

Glad to here that ;) Now I know the patch works on your box at least ;)

Quote:

Originally Posted by Bruce Hill (Post 3653095)

Thanks, and I too think you should send it to PiterPUNK. It's still his choice
what to do, and probably won't happen until Slackware-13.0 comes out.

Yes, I mailed Piter. But it seems Piter is out for a vacation or something like that? I sent "search for .new file" patch to him weeks ago but don't get his reply yet...

niels.horn

08-21-2009 11:41 PM

Worked very well here!
Slackpkg was slow on my virtual machines especially, but after this patch it's a lot faster.

Thanks!

Bruce Hill

08-22-2009 12:03 AM

rsync doesn't have to go through the firewall, if you just use it local -- which I do
for everything where the files are important. Maybe you have better bandwidth in
Beijing that I do in a village here in Dali. Heck, the phone wires coming in to my
house to use the ADSL modem look like chicken intestines, so I doubt the copper
quality is very good. And they're hung all over roofs, trees, etc.

PiterPUNK most probably wouldn't submit any changes until after Slackware-13.0 anyway.

Pat surely has a freeze on anything that's not broken, and important.

Now you have niels.horn (also in PiterPUNK's area) using it with good results.

PiterPUNK really does appreciate us using and submitting suggestions for slackpkg.

Chuck56

08-22-2009 07:52 AM

Quote:

Originally Posted by grissiom (Post 3653058)

Although I haven't tested it so much (I reverted two packages and "new slackpkg" can find them) but I think it would have no regressions. Any feedback is strongly welcome.

Feedback:

I am updating a desktop machine expecting to see the 8/20 thunderbird-2.0.0.23 patch from an LAN mirror via NFS. I replaced the 1 line (-) with the 5 lines (+) in /usr/libexec/slackpkg/core-functions.sh and got the following response.

Code:

root@slacker:~# slackpkg upgrade-all



Looking for packages to upgrade. Please wait... cut: /tmp/slackpkg.BK9mE8/pkglist: No such file or directory

DONE



No packages match the pattern for upgrade. Try:



        /usr/sbin/slackpkg install|reinstall





root@slacker:~#

When I backed out the changes it worked as expected. Did I do something wrong?

Bruce Hill

08-22-2009 08:00 AM

Please post the output of:

Code:

diff -u /usr/libexec/slackpkg/core-functions.sh /usr/libexec/slackpkg/core-functions.sh~

where /usr/libexec/slackpkg/core-functions.sh is the new file and /usr/libexec/slackpkg/core-functions.sh~ the old one.

Chuck56

08-22-2009 08:10 AM

Quote:

Originally Posted by Chuck56 (Post 3653468)

Did I do something wrong?

Self-inflicted wound...

I'm running slackpkg-2.70.5-noarch-1, not slackpkg-2.80.2-noarch-3. My modified feedback is that it doesn't appear to work in the stock 12.2 slackpkg.

grissiom

08-22-2009 10:35 AM

Quote:

Originally Posted by Bruce Hill (Post 3653216)

PiterPUNK most probably wouldn't submit any changes until after Slackware-13.0 anyway.

Pat surely has a freeze on anything that's not broken, and important.

Yes, it's reasonable.

Quote:

Now you have niels.horn (also in PiterPUNK's area) using it with good results.

PiterPUNK really does appreciate us using and submitting suggestions for slackpkg.

Yes, I think every developer's dream is getting people's suggestions and patches. Maybe he is busy this time. But I don't think he will miss it. Never mind~;-)

grissiom

08-22-2009 10:42 AM

Quote:

Originally Posted by Chuck56 (Post 3653468)

Code:

root@slacker:~# slackpkg upgrade-all



Looking for packages to upgrade. Please wait... cut: /tmp/slackpkg.BK9mE8/pkglist: No such file or directory

DONE



No packages match the pattern for upgrade. Try:



        /usr/sbin/slackpkg install|reinstall





root@slacker:~#

When I backed out the changes it worked as expected. Did I do something wrong?

Thanks~ So I know the patch won't work with slackpkg-2.70.5-noarch-1. Make it to generate pkglist and apply my patch is just painful...

gegechris99

08-24-2009 03:38 PM

Hello Grissiom,

I've used your patch and I'd like to report that "slackpkg upgrade-all" detected correctly the new upgraded packages for my -current machine.

And also, it was way much faster :)

Thank you.

BrZ	08-30-2009 08:17 AM

Thanks grissiom, worked fine on current 32 and 64.

gegechris99

09-14-2009 02:12 PM

@ grissiom,

I'm sorry but it seems that your patch does not work for security updates to a stable release.

In my case, I used mirror: http://slackware.osuosl.org/slackware-13.0/.

Your patch failed to detect the 3 patches since release of 13.0 (namely, mozilla-firefox, seamonkey and seamonkey-solibs).

Could the reason be that file PACKAGES.TXT is not updated for a stable release (it's rather patches/PACKAGES.TXT) contrary to -current?

grissiom

09-15-2009 12:43 AM

Quote:

Originally Posted by gegechris99 (Post 3682568)

Could you attach the output of "grep seamonkey-solibs /var/lib/slackpkg/pkglist"? Thanks!

gegechris99

09-15-2009 02:55 AM

Hello Grissiom,

I've upgraded the packages yesterday: I removed your patch and then slackpkg correctly showed the 3 packages to upgrade.

Do you still want the result of "grep seamonkey-solibs /var/lib/slackpkg/pkglist"?

If so, I'll post it tonight as I'm not at home at the moment :)

niels.horn

09-15-2009 09:04 AM

@Grissiom,

Just another feedback: I am using the patched version on -current and that picked up the packages to upgrade.
I have not patched SlackPkg on my 'stable' machine.

grissiom

09-15-2009 09:31 AM

Quote:

Originally Posted by gegechris99 (Post 3683242)

Yes, I still want the result of "grep seamonkey-solibs /var/lib/slackpkg/pkglist". Could you please also attach the result of "ls -l /var/lib/slackpkg/pkglist"? I still could not figure out what triggered the bug...

grissiom

09-15-2009 09:32 AM

Quote:

Originally Posted by niels.horn (Post 3683623)

@Grissiom,

Just another feedback: I am using the patched version on -current and that picked up the packages to upgrade.
I have not patched SlackPkg on my 'stable' machine.

Thanks for your feedback. I run -current and there is nothing wrong as well ;) But I don't have a 'stable' system installed..

Bruce Hill

09-15-2009 10:14 AM

grissiom,

I have replied twice, and both times I got some message about
my token has expired and press the back button. There was nothing
here, so I refreshed and posted again ... same result. I am too
tired to focus correctly, so I will just post some output and you
can evaluate it and we'll go from there tomorrow:

Code:

root@silas64:~# slackpkg update



Updating the package lists...

        Downloading...      

                        Copying ChangeLog.txt...



                No changes in ChangeLog.txt between your last update and now.

                Do you really want to download all other files (y/N)? y      



                List of all files

                        Copying patches/MANIFEST.bz2...

                        Copying slackware64/MANIFEST.bz2...

                        Copying extra/MANIFEST.bz2...      

                        Copying pasture/MANIFEST.bz2...    

                        Copying testing/MANIFEST.bz2...    

                Checksums                                  

                        Copying CHECKSUMS.md5...          

                        Copying CHECKSUMS.md5.asc...



                WARNING: Your mirror appears incomplete and is missing the

                        CHECKSUMS.md5 file. We recommend you change your mirror

                        so that package integrity can be verified against

                        CHECKSUMS.md5.



                Package List

                        Copying FILELIST.TXT...

grep: /tmp/slackpkg.M65RB3/CHECKSUMS.md5: No such file or directory



                ERROR: No CHECKSUMS.md5 and no FILELIST.TXT.

                      We strongly recommend that you change your mirror

                      to prevent security problems.



root@silas64:~# grep 'file:' /etc/slackpkg/mirrors

file://server2/ftp/pub/Linux/Slackware/slackware64-13.0/



root@silas64:~# grep 'CHECKSUMS.md5' /server2/ftp/pub/Linux/Slackware/slackware64-13.0/patches/



root@silas64:~# grep 'FILELIST.TXT' /server2/ftp/pub/Linux/Slackware/slackware64-13.0/patches/



root@silas64:~# ls /server2/ftp/pub/Linux/Slackware/slackware64-13.0/patches/

CHECKSUMS.md5  CHECKSUMS.md5.asc  FILE_LIST  MANIFEST.bz2  PACKAGES.TXT  packages/  source/



root@silas64:~# ls /server2/ftp/pub/Linux/Slackware/slackware64-13.0/patches/packages/

mozilla-firefox-3.5.3-x86_64-1.txt  mozilla-firefox-3.5.3-x86_64-1.txz.asc  seamonkey-1.1.18-x86_64-1.txz      seamonkey-solibs-1.1.18-x86_64-2.txt  seamonkey-solibs-1.1.18-x86_64-2.txz.asc

mozilla-firefox-3.5.3-x86_64-1.txz  seamonkey-1.1.18-x86_64-1.txt          seamonkey-1.1.18-x86_64-1.txz.asc  seamonkey-solibs-1.1.18-x86_64-2.txz



root@silas64:~# grep seamonkey-solibs /var/lib/slackpkg/pkglist

slackware64 seamonkey-solibs 1.1.17 x86_64 1 seamonkey-solibs-1.1.17-x86_64-1 ./slackware64/l txz



root@silas64:~# diff -u /usr/libexec/slackpkg/core-functions.sh /usr/libexec/slackpkg/core-functions.sh~

--- /usr/libexec/slackpkg/core-functions.sh    2009-08-22 09:08:24.091148641 +0800

+++ /usr/libexec/slackpkg/core-functions.sh~    2009-07-27 09:33:49.000000000 +0800

@@ -567,11 +567,7 @@

                        done

                ;;

                upgrade-all)

-                      cut -f2-6 -d\  ${TMPDIR}/tmplist > ${TMPDIR}/lpkg

-                      cut -f2-6 -d\  ${TMPDIR}/pkglist > ${TMPDIR}/spkg

-                      potential=$(cat ${TMPDIR}/{s,l}pkg | sort | uniq -u | cut -f1 -d\  - | uniq -d )

-                      #for i in $(cut -f2 -d\  ${TMPDIR}/tmplist); do

-                      for i in $potential; do

+                      for i in $(cut -f2 -d\  ${TMPDIR}/tmplist); do



                                givepriority ${i}

                                [ ! "$FULLNAME" ] && continue

grissiom

09-15-2009 11:47 AM

Finally worked out a patch for this problem. But I have to admit that this patch is "stupid and ugly". So if anyone there have a better idea, please tell me.

The "patch":
please edit your /usr/libexec/slackpkg/core-functions.sh around line 569 to something like this:

Code:

  568                ;;

  569                upgrade-all)

  570                        cut -f2-6 -d\  ${TMPDIR}/tmplist > ${TMPDIR}/lpkg

  571                        cut -f2-6 -d\  ${TMPDIR}/pkglist > ${TMPDIR}/spkg

  572                        grep -E '^patches' ${TMPDIR}/pkglist | cut -f2-6 -d' ' > ${TMPDIR}/ppkg

  573                        # use packages in patches mask anther packages.

  574                        for i in $(cut -f1 -d' ' ${TMPDIR}/ppkg); do

  575                                sed -i -e "/^$i /d" ${TMPDIR}/spkg

  576                        done

  577                        # bring the patches back

  578                        cat ${TMPDIR}/ppkg >> ${TMPDIR}/spkg

  579                        potential=$(cat ${TMPDIR}/{s,l}pkg | sort | uniq -u | cut -f1 -d\  - | uniq -d )

  580                        #for i in $(cut -f2 -d\  ${TMPDIR}/tmplist); do

  581                        for i in $potential; do

Hope this time all would be Ok ;)

grissiom

09-15-2009 11:49 AM

@Bruce Hill

I think you need to change your mirror as there are so many warnings ;) Thanks for your help anyway ;)

gegechris99

09-15-2009 03:26 PM

Hello Grissiom,

Thank you for the new patch. I'll test it when the next 13.0 security update occurs.

For the record, here are the data you requested:

Code:

chris@darkstar ~ $ grep seamonkey-solibs /var/lib/slackpkg/pkglist

patches seamonkey-solibs 1.1.18 i486 2 seamonkey-solibs-1.1.18-i486-2 ./patches/packages txz

slackware seamonkey-solibs 1.1.17 i486 1 seamonkey-solibs-1.1.17-i486-1 ./slackware/l txz

chris@darkstar ~ $ ls -l /var/lib/slackpkg/pkglist

-rw-r--r-- 1 root root 86135 2009-09-14 20:44 /var/lib/slackpkg/pkglist

grissiom

09-15-2009 06:58 PM

Quote:

Originally Posted by gegechris99 (Post 3684092)

Hello Grissiom,

Thank you for the new patch. I'll test it when the next 13.0 security update occurs.

For the record, here are the data you requested:

Code:

chris@darkstar ~ $ grep seamonkey-solibs /var/lib/slackpkg/pkglist

patches seamonkey-solibs 1.1.18 i486 2 seamonkey-solibs-1.1.18-i486-2 ./patches/packages txz

slackware seamonkey-solibs 1.1.17 i486 1 seamonkey-solibs-1.1.17-i486-1 ./slackware/l txz

chris@darkstar ~ $ ls -l /var/lib/slackpkg/pkglist

-rw-r--r-- 1 root root 86135 2009-09-14 20:44 /var/lib/slackpkg/pkglist

Ok. Thank you for testing my patch ;-)

gegechris99

09-20-2009 10:37 AM

Hello Grissiom,

I tested your latest patch on Slackware 13.0 with the security update of 19th September. It works :)

Thanks again.

I hope it somehow will make it into a future version of slackpkg.

All times are GMT -5. The time now is 10:18 AM.