slackware activity By The Numbers

Pithium · 11-23-2020, 09:57 PM

Hi All,

I've been working on a fun project that I'd like to share with the class. Something I've run into issues with as a slackware user is an effective way to communicate the "aliveness" of the project. For users new to Slackware or Linux in general it can be difficult to simply refer them to the changelog. You also need to aggregate all active changelogs, not just -current.

So...I decided to write some python scripts to parse and analyze the data contained in all slackware changelogs (-current and older versions) in order to provide myself with numerical talking points. The scripts are still fragile, however I have reached the point where I'm semi confident in the numbers I've got and would like to share that info with everyone. If you got this far and are interested in slackware package statistics, read on!

I'm counting the number of changes PV makes to slackware on a per package basis. Every time Pat adds, removes, upgrades, or otherwise changes a package he notes this in the changelog and datestamps it. I parse this into an sqlite file and then query that on the basis of version, timeframe, etc. As an example, here is a raw count of all changes made to a given slackware version from 12.0 onwards..

Code:

12	  1783
12.1      1410
12.2      964
13	  3909
13.1      3451
13.37     4134
14	  5237
14.1      7395
14.2      8867
-current  26514

( this is based on the changelogs as of November 23, 2020 )

For those of you who are more visual, I've attached a graph of the same data made in libreoffice. You could say that these numbers represent Pat's workload in totality, since all of these updates go through him at some point in slackware's process. I plan to release a more comprehensive analysis of slackware dev activity once I'm more confident in my parsing scripts. The general idea is to have a way to "advertise" the work being done in a simple and factual way.

If anyone has any thoughts let me know. In particular if there are any specific trends you guys would like to look at I'm looking for ideas.

upnort · 11-24-2020, 12:25 AM

Interesting! I think somehow the time between releases should be considered too.

chrisretusn · 11-24-2020, 01:00 AM

Quote:

Originally Posted by upnort

Interesting! I think somehow the time between releases should be considered too.

Agree and the graph should be normalized.

igadoter · 11-24-2020, 03:22 AM

For me -current has not much sense. I mean -current is always -current from beginning of time. How old is -current? These numbers as well may reflect changes in upstream. 12.2 was last Slackware with KDE 3.5. I think more interesting would be follow lifetime of package. I mean those which are beyond a/ and required. Say in upstream software is long dead - but still exists in Slackware. Numbers may reflect maintainer activity or peak in stability - I always considered 12.2 as one of most solid release. But 13.0 was first with KDE 4.0. Just thinking about is painful.

FTIO · 11-24-2020, 04:57 AM

I would imagine any big jumps in differences of numbers would mean some kind of huge change in a piece of software somewhere...such as KDE4 to KDE5 would be *huge* (and I think that shows in the numbers somehow there), or Python 2 to Python 3, use ALSA again or continue with <shudder> pulseaudio or figure out a way to offer either as a choice, X or Wayland, etc, etc. Would things like this skew anything anyhow with the numbers?

FTIO · 11-24-2020, 05:05 AM

Quote:

Originally Posted by igadoter

For me -current has not much sense. I mean -current is always -current from beginning of time. How old is -current? These numbers as well may reflect changes in upstream. 12.2 was last Slackware with KDE 3.5. I think more interesting would be follow lifetime of package. I mean those which are beyond a/ and required. Say in upstream software is long dead - but still exists in Slackware. Numbers may reflect maintainer activity or peak in stability - I always considered 12.2 as one of most solid release. But 13.0 was first with KDE 4.0. Just thinking about is painful.

I still have the CD's I burned after downloading - via a nice, speedy landline (took forever!! lol) - of one of the 12's. I sure do miss KDE3. It was fast and 'just worked', and looked better too IMHO. I can't honestly say though if it ran better/longer/more trouble free than KDE4 on the 13's or my 14.1 x86 or 14.2 x86_64. I *am* currently starting to have freeze-ups the last week or so on the 14.2, but I'm betting on it being the SSD starting to take a dive.

drumz · 11-24-2020, 08:39 AM

I think separate tallies for "Added", "Removed", "Upgraded", and "Rebuilt" would be interesting, as well as what you already have (changed = added + removed + upgraded + rebuilt).

Also I like the idea of normalizing the data with respect to time. Maybe changes/week would give "nice" numbers? I think changes/day would give numbers too small and abstract.

bassmadrigal · 11-24-2020, 11:03 AM

What terms were you searching for to get your numbers? Because I used grep and wc to count all the entries of "Upgraded. Added. Removed. Rebuilt." against -current's changelog and I got 13132 changes.

Code:

jbhansen@craven-moorhead:~$ head -n1 /share/gothrough/slackware-mirrors/slackware64-current/ChangeLog.txt        
Mon Nov 23 21:29:14 UTC 2020
jbhansen@craven-moorhead:~$ grep -e Upgraded\\. -e Added\\. -e Removed\\. -e Rebuilt\\. /share/gothrough/slackware-mirrors/slackware64-current/ChangeLog.txt | wc -l
13132
jbhansen@craven-moorhead:~$ grep Upgraded\\. /share/gothrough/slackware-mirrors/slackware64-current/ChangeLog.txt | wc -l
8063
jbhansen@craven-moorhead:~$ grep Added\\. /share/gothrough/slackware-mirrors/slackware64-current/ChangeLog.txt | wc -l
557
jbhansen@craven-moorhead:~$ grep Removed\\. /share/gothrough/slackware-mirrors/slackware64-current/ChangeLog.txt | wc -l
111
jbhansen@craven-moorhead:~$ grep Rebuilt\\. /share/gothrough/slackware-mirrors/slackware64-current/ChangeLog.txt | wc -l
4401

If my number is accurate and you want to figure out how many packages per day were accomplished on average, you can do that with the following command (replacing the two locations to the changelog to where it is on your system). I imagine this can probably be simplified, but it takes the count from the changelog I used above, and then divides that by the total number of days since the changelog was started (takes today in epoch form (number of seconds), and then takes the date of the first changelog entry in epoch form and subtracts the latter from the former, and then divides the resultant number of seconds by 60 seconds in a minute, 60 minutes in an hour, and 24 hours in a day to get the total number of days).

Code:

jbhansen@craven-moorhead:~$ echo $(grep -e Upgraded\\. -e Added\\. -e Removed\\. -e Rebuilt\\. /share/gothrough/slackware-mirrors/slackware64-current/ChangeLog.txt | wc -l) / $(( (($(date '+%s') - $(date -d "$(grep -a1 "+--" /share/gothrough/slackware-mirrors/slackware64-current/ChangeLog.txt | tail -n1)" '+%s') ) / (60*60*24)) )) | bc -l | xargs printf "%.2f packages per day\n"
8.17 packages per day

We have to do a slight change for 14.2's calculation due to it being in maintenance mode and we don't want to count patches after it was released, just the development timeline.

Code:

jbhansen@craven-moorhead:~$ echo $(grep -e Upgraded\\. -e Added\\. -e Removed\\. -e Rebuilt\\. /share/gothrough/slackware-mirrors/slackware64-14.2/ChangeLog.txt | wc -l) / $(( (($(date -d "$(grep -B1 "14.2 x86_64" /share/gothrough/slackware-mirrors/slackware64-14.2/ChangeLog.txt | head -n1)" '+%s') - $(date -d "$(grep -A1 "+--" /share/gothrough/slackware-mirrors/slackware64-14.2/ChangeLog.txt | tail -n1)" '+%s') ) / (60*60*24)) )) | bc -l | xargs printf "%.2f packages per day\n"
4.50 packages per day

Keep in mind, -current has had a mass rebuild, and "Rebuilt." comprises of about 1/3 of the entries. If we remove "Rebuilt." entries, then the number of packages per day goes down to 5.43 (which is still higher than 14.2).

0XBF · 11-24-2020, 12:03 PM

I think the information may be more meaningful if its averaged over some time, like bass was doing with per day. If the x axis of the graph was months or years, since releases are different amounts of time between them. You could still overlay the releases over that to get some context.

Neat idea, too bad it can't predict the release of 15

Pithium · 11-24-2020, 12:41 PM

Quote:

Originally Posted by drumz

I think separate tallies for "Added", "Removed", "Upgraded", and "Rebuilt" would be interesting, as well as what you already have (changed = added + removed + upgraded + rebuilt).

Also I like the idea of normalizing the data with respect to time. Maybe changes/week would give "nice" numbers? I think changes/day would give numbers too small and abstract.

I'm actually already recording that. Initially I ran into the same problem bassmadrigal did when searching for certain keywords. What I do is look for instances of the pattern $PKG: $REASON and try to use whatever Pat put into the log rather than decide for myself what it should be. Problem was, this has slowly changed over time so I can't actually rely on $REASON to be consistent from 2007 to now. This is largely the reason why I'm not going back further than 12.0.

Every update in the changelog is broken down into these datapoints:

Code:

arch,version,timestamp,patched,upgraded,updated,moved,added,removed,fixed,rebuilt,other

So I can query this and pull the number of upgrades for -current.

Code:

9095 for x86
8040 for x86_64
17135 upgrades total

But if we look at pkgs added we get:

Code:

536 x86
528 x86_64
1064 total

But every once and a while Pat decides to throw a curve ball and use the word "Updated". I'm not entirely sure what constitutes an 'update' versus an 'upgrade' and it only came up 80 times since the start of 12.0. This is also an area where my code might be breaking so it's possible things aren't being counted as intended.

Code:

2 x86
2 x86_64
4 total

Attached another graph of the change "types". My next step is to start breaking all this down per timeframe since lifetime numbers are not the most useful. I'm thinking monthly would be good but just to poke fun at the business management types I might go for quarterly reports

bassmadrigal · 11-24-2020, 02:44 PM

Now I understand you're counting both 32 and 64bit development, which I didn't think about (and will skew the numbers from 12.2 and lower as those versions were only developed for 32bit). I'd be curious if your code provides the same number of changes for the 64bit version of -current as mine does (13,134 changes).

Based on your comments, I found there were 2 entries of "Updated." referring to SlackBuilds being updated without providing upgraded packages that weren't accounted for in my original command. I also wasn't sure this accounted for everything, so I revamped my test:

Code:

 grep -e ^a/ -e ^ap/ -e ^d/ -e ^e/ -e ^f/ -e ^k/ -e ^kde/ -e ^kdei/ -e ^l/ -e ^n/ -e ^t/ -e ^tcl/ -e ^x/ -e ^xap/ -e ^xfce/ -e ^y/ -e ^testing/ -e ^extra/ -e ^isolinux/ -e ^kernels/ -e ^pasture/ -e ^usb-pxe-installers/  /location/to/slackware64-current/ChangeLog.txt | cut -d' ' -f3 | sort | uniq -c

This gives:

Code:

    557 Added.
   4401 Rebuilt.
    111 Removed.
      2 Updated.
   8063 Upgraded.

This could probably be automated further to build the package sets automatically based on the folder structure of the slackware{,64} directory on the installation media to account for any changes in set names, but you'd need to have the full directory available and I'm too lazy to code it.

We can inverse this to find out how many updates each package set saw (256 kernel entries and I didn't realize there weren't any updates for f/ or kdei/ -- testing/ saw an outstanding 1300+ updates, mainly due to PAM and vtown being in there for a bit).

Code:

grep -e Added\\. -e Rebuilt\\. -e Removed\\. -e Upgraded\\. -e Updated\\.  /share/gothrough/slackware-mirrors/slackware64-current/ChangeLog.txt | cut -d/ -f1 | sort | uniq -c

Code:

   1872 a
    836 ap
   1335 d
     18 e
    195 extra
    279 isolinux
    264 k
    337 kde
    265 kernels
   2578 l
   1476 n
      5 pasture
     34 t
     23 tcl
   1306 testing
    279 usb-and-pxe-installers
   1289 x
    594 xap
    147 xfce
      2 y

Pithium · 11-24-2020, 05:43 PM

Looks like we are getting the same results after I re-downloaded all the changelogs. Also have some of the basic groundwork down for trending changes. A running total for each month of 2020 (so far) for all changes made to the Slackware tree. Still need to automate it but that's just time on my end. You can see that there was a massive spike where vtown was added.

All I have to do now is set it up so I can specify a given timeframe, and it returns the data for that timeframe!

Pithium · 01-22-2021, 12:28 PM

Hey all, don't mean to resurrect an old thread but if anyone was interested I've made significant progress on the charting. Moved to gnuplot for graphing and cleaned some other stuff up in how I'm looking at the data.

I'm also now comparing package changes to the overall size of the distro, over time. future updates will go on my blog since I can't embed images here. Attached is summary of all updates since 12.0 till now.

astrogeek · 01-22-2021, 01:15 PM

Now that is more interesting that recent pandemic graphs I have seen - it has context - thanks!

The relative continuity of trend of the solid blue line since 2016 speaks volumes!

Pithium · 01-22-2021, 01:26 PM

It's pretty clear that while the gap between stable releases has grown, so has the amount of monthly changelog activity.