current pkgtool performance

lazardo · 01-29-2022, 02:10 PM

Updated after numerator/denominator swap

When upgrading current after +45 days, it appears installation time has gone up dramatically from general observations since slamd-64 days

There are earlier reports detailing the new upgradepkg work flow, but no actual measurements. So.

153 packages upgraded, 53 new installs.
53 minutes total, 0.25 minutes per package @1.49 load average

* asus 1215N laptop
* intel SSD (208MB/s read, 180MB/s write)
* 64bit quad intel Atom D525 @3600 Bogos/core (no power management, always full throttle)

Observed in htop tree view:
* 4G system memory fully utilized
* apparent parallelism
*xz decompress uses all cores.

note: slackpkg not used, just installpkg

In DaysOfYore, start a job and go get coffee without a thought...

hitest · 01-29-2022, 04:29 PM

Your performance seems okay given your system specs.

My lowest spec Slackware64-current unit has 8 GB RAM and is an i5 2.40 GHz T420 Thinkpad. It's still usable. My i5 2.40 Ghz T410 Thinkpad has 4 GB RAM; I run OpenBSD on that unit.

GazL · 01-29-2022, 05:02 PM

I think it's the process of checking for files to be removed that is the issue. I noticed upgradepkg got significantly slower when the "meltdown" workarounds went into the kernel. I suspect the process involves lots of context switches. The issue is especially noticeable on slow spinning rust discs that non-ssd laptops tend to have.

My CRUX system's package operations are lightning fast in comparison (even on the same hardware), but it follows the "only one package can provide a specific file" model, so it doesn't need to do this sort of duplicate checking.

Didier Spaier · 01-29-2022, 05:46 PM

spkg is way faster. Caveat: it doesn't do a re-installation when upgrading a package, which can cause issues. As an aside upgradepkg *can* do the same if NOT_PARANOID is set to true, but probably few users know this "feature", let alone use it.

lazardo · 01-29-2022, 07:35 PM

I saw NOT_PARANOID, will use.

Perhaps the current upgrade cycle, versus not-current, makes the illusion of time

Code:

j u   s    t       s       l               o                                                 .

Didier Spaier · 01-30-2022, 02:20 AM

Quote:

Originally Posted by lazardo

I saw NOT_PARANOID, will use.

At your own risks

And be aware that what takes most of the time in case of big packages like kernel-source is checking if each file is also shipped in another package, so do not expect a major speed increase. A shell script can't compete with a compiled program wrt speed, and bash is not the faster interpreter. Fortunately we don't spend most of the day upgrading big packages.

volkerdi · 01-31-2022, 02:23 PM

Quote:

Originally Posted by Didier Spaier

spkg is way faster. Caveat: it doesn't do a re-installation when upgrading a package, which can cause issues. As an aside upgradepkg *can* do the same if NOT_PARANOID is set to true, but probably few users know this "feature", let alone use it.

I'd be curious to see this benchmarked against the default upgradepkg. I suspect that it doesn't shave off as much time (if any) as it used to, since upgradepkg no longer runs the install script during the preinstall phase, and when running installpkg the second time (with the installation script) no longer needlessly reinstalls all the files, only ones that may have gone missing after the removepkg.

I'd really recommend against using NOT_PARANOID. There are simple test cases where it leads to a broken package installation.

Pithium · 01-31-2022, 03:42 PM

Probably not strictly related.. but one thing I've found is that packages with a very large number of files tend to take a long time when uninstalling. After some experimentation I found that this was due to a limit in the render speed of the console I was using (in this case Konsole).

Turns out that the speed at which Konsole can spit out text is slower than the actual rate at which the operation runs and became a bottle neck when thousands of lines are being printed repeatedly printed to the screen. I guess the lesson is to rule out any issues with your terminal emulator slowing down the task. This is especially noticeable with both kernel-source and qt5.

Didier Spaier · 01-31-2022, 05:41 PM

Quote:

Originally Posted by volkerdi

I'd be curious to see this benchmarked against the default upgradepkg.

Results below, Sir.

The target of each command is displayed before the results. In all cases I have redirected stdout to /dev/null, so that the display speed on the terminal do not influence the results. As expected, the most discriminant factor is the number of files to remove.

PS This is on Slackware64-current up to date at time of writing.

Code:

With spkg -u        With upgradepkg
============        ===============

MPlayer-20200103-x86_64-4.txz
real    0m2,057s    0m4,626s
user    0m1,134s    0m3,826s
sys     0m0,275s    0m1,208s

NetworkManager-1.32.10-x86_64-4.txz
real    0m2,138s    0m4,934s
user    0m1,054s    0m4,392s
sys     0m0,272s    0m1,421s

kernel-generic-5.15.17-x86_64-1.txz
real    0m0,748s    0m3,245s
user    0m0,569s    0m2,726s
sys     0m0,093s    0m0,972s

kernel-modules-5.15.17-x86_64-1.txz
real    0m5,174s    0m42,325s
user    0m5,038s    0m34,116s
sys     0m1,205s    0m14,474s

kernel-source-5.15.16-noarch-2.txz
real    0m36,608s    5m0,923s
user    0m14,315s    2m15,774s
sys     0m7,689s    1m52,735s

mariadb-10.6.0-x86_64-1.txz
real    0m18,505s    0m21,638s
user    0m4,042s    0m20,729s
sys     0m1,198s    0m6,869s

vim-8.2.4212-x86_64-1.txz
real    0m6,215s    0m7,548s
user    0m1,260s    0m6,678s
sys     0m0,473s    0m2,031s

xfsprogs-5.12.0-x86_64-1.txz
real    0m1,012s    0m4,752s
user    0m0,611s    0m2,665s
sysa    0m0,081s    0m0,817s

Total
real    1m12,465s    6m29,998s

volkerdi · 01-31-2022, 06:31 PM

Quote:

Originally Posted by Didier Spaier

Results below, Sir.

I was actually wondering about NOT_PARANOID vs. standard upgradepkg but realize now that I was completely unclear about that. ;-)

Didier Spaier · 02-01-2022, 03:52 AM

Quote:

Originally Posted by volkerdi

I was actually wondering about NOT_PARANOID vs. standard upgradepkg but realize now that I was completely unclear about that. ;-)

The difference is not significant for kernel-source:

Code:

--no-paranoid
real    4m14,555s
user    2m25,865s
sys     2m17,907s

without option
real    4m10,889s
user    2m25,426s
sys     2m16,280s

Whether this option should be removed before the release of Slackware 15 is left to the reader as an exercise

PS during these benchmarks the system was not otherwise idle, so take the results with a pinch of salt.

gapan · 02-01-2022, 05:32 AM

For what it's worth, I benchmarked installing the complete slackware tree with installpkg and spkg. I'm on a fully updated slackware current installation.

Code:

# time installpkg --terse --root /mnt/nvme/test-inst-pkgtools/ */*.t[gx]z
...
real    10m33.184s
user    9m5.068s
sys     2m5.911s

# time spkg -q -q --root=/mnt/nvme/test-inst-spkg/ */*.t[gx]z

real    5m7.429s
user    3m51.285s
sys     0m35.258s

# time spkg --no-ldconfig --no-gtk-update-icon-cache -q -q --root=/mnt/nvme/test-inst-spkg/ */*.t[gx]z
real    4m37.470s
user    3m31.895s
sys     0m30.037s

lazardo · 02-01-2022, 04:49 PM

$TMP in tmpfs IF mem >= 4G ELSE slow_anyway

the3dfxdude · 02-01-2022, 09:04 PM

I think the upgrade from 14.2 to 15.0 is the one that really matters. The in-between slack-current package respins are mostly useless to me, and if I had to follow -current for some reason, I'd block the more annoying respins and let some of them go stale. It's not worth the bit churn if it's too slow for you.

If you ask me, the font packages are the slowest I've seen. I'd profile upgrading those if you really want to see slow. But the fact I'd only do this if needed, I don't know how much I'd care. In fact, I might go back and removepkg off some of them.

dchmelik · 02-01-2022, 09:49 PM

Will 15 have spkg since faster than installpkg?