google brotli - what you really want to know ;)

Compiled from git today. While some data store scenarios benefit by squeezing every last KB they are almost always 'compress once, read forever'. You probably don't want to swap out for brotli just yet. I was curious given all the somewhat misleading coverage.

Cheers,

MB		                                user	sys	elapsed	      seconds	sec/MB	MB/s	CPU	mem	options
593	libreoffice-4.1.6-x86_64-1alien									
197	libreoffice-4.1.6-x86_64-1alien.pigz1	21.24	0.40	00:00:10.57	11	0.06	53.91	204%CPU	34608	pigz -p2 -1
174	libreoffice-4.1.6-x86_64-1alien.pigz	51.10	0.47	00:00:51.63	52	0.30	11.40	99%CPU	9824	pigz -p1
139	libreoffice-4.1.6-x86_64-1alien.xz1	92.73	0.52	00:01:33.32	93	0.67	6.38	99%CPU	43632	xz -1
116	libreoffice-4.1.6-x86_64-1alien.xz	519.14	0.90	00:08:40.41	520	4.48	1.14	99%CPU	390016	default
121	libreoffice-4.1.6-x86_64-1alien.bro	4065.67	51.92	01:08:40.00	4120	34.05	0.14	99%CPU	3428512	default
										
Intel(R) Core(TM) i3 CPU M 330  @ 2.13GHz										
8GB DDR3@1066										
linux 3.18.21, slackware64 14.1, gcc 4.8.2										
										
test input: decompressed libreoffice slackbuild from alien bob										
sample command: /usr/bin/time -o xz1.time xz -v -1 > libreoffice-4.1.6-x86_64-1alien.xz1 < libreoffice-4.1.6-x86_64-1alien										
all I/O in tmpfs (memory file system)

http://google-opensource.blogspot.co...mpression.html
https://github.com/google/brotli

So all that noise these days for brotli was for nothing or such.

BTW, with "pixz" the XZ implementation is even more quicker, pixz is the fastest parallelized implementation of LZMA2 (works on top of xz-utils libraries), scales up to N cores both for encoding and decoding, plus the archives made remain in specification.

Originally Posted by man-walking So all that noise these days for brotli was for nothing or such.

The industry noise is unusually misleading given it is not the pied piper-esq solution that I've seen emphasized.

BTW, with "pixz" the XZ implementation is even more quicker, ...

Couldn't build pixz due to automake version mismatch so grabbed the multi-threaded v5.2.1 xz from -current:

MB		                                user	sys	elapsed	      seconds	sec/MB	MB/s	CPU	mem	options
139	libreoffice-4.1.6-x86_64-1alien.xz51	100.17	0.89	01:41.15	101	0.73	5.87	99.00%	43792	xz -1 -T1	v5.2.1
140	libreoffice-4.1.6-x86_64-1alien.xz52	107.18	0.41	00:53.70	54	0.39	10.98	200.00%	145792	xz -1 -T2	v5.2.1
140	libreoffice-4.1.6-x86_64-1alien.xz54	121.70	0.72	00:31.68	32	0.22	26.95	386.00%	276752	xz -1 -T4	v5.2.1
118	libreoffice-4.1.6-x86_64-1alien.xz54d	566.67	2.10	02:30.57	151	1.28	3.93	377%	150953	xz -T4		v5.2.1

The single thread was 8% slower than v5.0.5, but it does scale nicely. I'll try the multi-threaded xz on my DVD VOBs to see if the speed/compression is worth replacing pigz. ABs libreoffice package is a good mix of binary, text and small png images, fyi.

Cheers,

UPDATE:
VOBs remain difficult to compress, xz is not better.

               1206MB	nurse_jackie_S6_D2_4.VOB
xz   -0  -T4 == 1178MB(2.4%), 88sec
xz   -0e -T4 == 1173MB(2.8%), 111sec
pigz -6  -p4 == 1174MB(2.7%), 14sec

The comparison with XZ seems to have gotten a lot of emphasis but that is not the key point IMHO. Brotli's strength is in compression sizes and times for serving data on the web. Here it is competing with "Content-Encoding: gzip".

In my own very quick test, Brotli destroys GZip in compression size and times to compress and decompress, using the contents of Chrome 46.0.2490.71 for 64bit Linux as a source of files. The key is to set a compression quality level lower than the default.

First, I'll grab the latest Chrome and extract the cpio archive out of it:

$ wget -q https://dl.google.com/linux/chrome/rpm/stable/x86_64/google-chrome-stable-46.0.2490.71-1.x86_64.rpm
$ grep -abom1 7zXZ google-chrome-stable-46.0.2490.71-1.x86_64.rpm | cut -d: -f1
40081
$ tail -c+40081 google-chrome-stable-46.0.2490.71-1.x86_64.rpm | xz -d > google-chrome-stable-46.0.2490.71-1.cpio
$ ls -lh google-chrome-stable-46.0.2490.71-1.cpio
-rw-r--r-- 1 ruario users 172M Oct 19 09:00 google-chrome-stable-46.0.2490.71-1.cpio

Now, some comparison with GZip:

$ time gzip -9k google-chrome-stable-46.0.2490.71-1.cpio

real    0m32.913s
user    0m32.849s
sys     0m0.094s
$ time bro --quality 1 --input google-chrome-stable-46.0.2490.71-1.cpio > google-chrome-stable-46.0.2490.71-1.cpio.br

real    0m2.309s
user    0m2.235s
sys     0m0.075s
$ ls -lh google-chrome-stable-46.0.2490.71-1.cpio.{br,gz}
-rw-r--r-- 1 ruario users 62M Oct 19 09:03 google-chrome-stable-46.0.2490.71-1.cpio.br
-rw-r--r-- 1 ruario users 63M Oct 19 09:00 google-chrome-stable-46.0.2490.71-1.cpio.gz
$ time gzip -cd google-chrome-stable-46.0.2490.71-1.cpio.gz > /dev/null

real    0m1.577s
user    0m1.566s
sys     0m0.013s
$ time bro --decompress --input google-chrome-stable-46.0.2490.71-1.cpio.br > /dev/null

real    0m0.836s
user    0m0.815s
sys     0m0.021s

Note: This is a Lenovo Yoga 3 running Slackware-current. Not that that it matters too much as it is the relative times that are important. In addition, I have provided the commands I ran so you are free to retest on your own hardware.

A quick follow up comparing with BZip2, setting a quality level of 5 (so that compression level is close to BZip2's 9)

$ time bzip2 -9k google-chrome-stable-46.0.2490.71-1.cpio

real    0m22.209s
user    0m22.110s
sys     0m0.076s
$ time bro --quality 5 --input google-chrome-stable-46.0.2490.71-1.cpio > google-chrome-stable-46.0.2490.71-1.cpio.br

real    0m9.762s
user    0m9.665s
sys     0m0.107s
$ ls -lh google-chrome-stable-46.0.2490.71-1.cpio.{br,bz2}
-rw-r--r-- 1 ruario users 56M Oct 19 09:09 google-chrome-stable-46.0.2490.71-1.cpio.br
-rw-r--r-- 1 ruario users 58M Oct 19 09:00 google-chrome-stable-46.0.2490.71-1.cpio.bz2
$ time bzip2 -cd google-chrome-stable-46.0.2490.71-1.cpio.bz2 > /dev/null

real    0m8.765s
user    0m8.749s
sys     0m0.024s
$ time bro --decompress --input google-chrome-stable-46.0.2490.71-1.cpio.br > /dev/null

real    0m0.799s
user    0m0.789s
sys     0m0.011s

You will note that even with quality 5, Brotli is still out performing GZip from my previous test in all regards!

Following up to compare with XZ for the sake of it and indeed for this sample of data, Brotli cannot match the compression size of XZ and takes a relatively long time to complete. However, this is not the type of data Brotli has been optimised for since it contains large binaries, while Brotli is designed to work with text files like HTML, CSS and JavaScript. In addition, while compression is slower you would expect this to happen once, whereas the data would be decompressed by the receiving clients (browsers) multiple times. Hence decompression times should be strongly favoured. As you can see Brotli is much faster than XZ at decompressing this collection of files on my system (In fact even at quality 11, it is still faster than GZip):

$ time xz -9k google-chrome-stable-46.0.2490.71-1.cpio

real    1m52.666s
user    1m52.081s
sys     0m0.686s
$ time bro --quality 11 --input google-chrome-stable-46.0.2490.71-1.cpio > google-chrome-stable-46.0.2490.71-1.cpio.br

real    13m14.687s
user    13m9.831s
sys     0m5.521s
$ ls -lh google-chrome-stable-46.0.2490.71-1.cpio.{br,xz}
-rw-r--r-- 1 ruario users 48M Oct 19 09:49 google-chrome-stable-46.0.2490.71-1.cpio.br
-rw-r--r-- 1 ruario users 45M Oct 19 09:00 google-chrome-stable-46.0.2490.71-1.cpio.xz
$ time xz -cd google-chrome-stable-46.0.2490.71-1.cpio.xz > /dev/null

real    0m4.054s
user    0m4.023s
sys     0m0.035s
$ time bro --decompress --input google-chrome-stable-46.0.2490.71-1.cpio.br > /dev/null

real    0m0.955s
user    0m0.936s
sys     0m0.020s

In summary, no Brotli is not some wonder compressor that beats XZ in all cases but I suspect that it probably better than XZ for its intended use case (good compression of text files and with very quick decompression). It is also a better general compressor when comparing with GZip and BZip2. It could also be interesting in packaging. Consider that SBo packages are usualy created as .tgz for speed. Using .tbr with quality level 1 through 5 would be faster and result in smaller packages. You might even find a case for using Brotil for packages on Linux install media using quality level 11. You would get something close to XZ in terms of absolute size but install should be quicker because of vastly better decompression times.

Originally Posted by ruario You might even find a case for using Brotil for packages on Linux install media using quality level 11. You would get something close to XZ in terms of absolute size but install should be quicker because of vastly better decompression times.

If I may digress a bit: maybe using a package manager like slpkg can bring even higher speeds that changing the compression app? But these two ways are not mutually exclusive, I assume.

Originally Posted by man-walking So all that noise these days for brotli was for nothing or such.

Not for nothing. It is a great compressor and appears to handily beat GZip and BZip2 in terms of size and speed. These might not be "cutting edge" compression formats these days but they are widespread, particularly gzip for content encoding on the web and Brotli is aimed at replacing some of that. XZ/LZMA could have been a contender for this if it wasn't so slow at decompression. GZip/Deflate's decompression speed (with "relatively" good compression) is the reason why it is still widespread. Zopfli was interesting in the industry because it achieved a minor improvement in compression, whilst retaining good decompression speed. If you have ever tried it, you will know that Zopfli takes an "age" to compress data but few care because the end size and decompression speed are the points most cared about for use on the web. Brotli destroys Zopfli for this specific use case.

Originally Posted by lazardo The industry noise is unusually misleading given it is not the pied piper-esq solution that I've seen emphasized.

I think that the problem is reporters wanting a juicy headline. Brotli is a very decent compression format for its intended use case.

Originally Posted by Didier Spaier If I may digress a bit: maybe using a package manager like slpkg can bring even higher speeds that changing the compression app? But these two ways are not mutually exclusive, I assume.

Indeed, they are not mutually exclusive at all. Also I wasn't just thinking about Slackware. XZ is a currently the "gold standard" for package compression on all distros. It results in small files that are relative quick to decompress. Brotli is almost as small and much quicker to decompress. It would be interesting to see if any of the distros consider it for packaging.

This got me interested so I did some tests. I compared all different quality settings for gzip, xz (default versions in slackware 14.1) and brotli (version 0.2.0).

Ran the tests on three files:
1. A gslapt package (~ 760K uncompressed)
2. A package of brotli itself (~ 1.9MB uncompressed)
3. A firefox package (~ 87MB uncompressed)

Here's what I found. Sizes are in KB, times in seconds. All tests were made on an Intel NUC with an i3 CPU and slackware64 14.1.

gslapt compression:

Comp.   Size    Time
gzip -1 228     0.01
gzip -2 220     0.01
gzip -3 212     0.02
gzip -4 188     0.02
gzip -5 184     0.03
gzip -6 184     0.04
gzip -7 180     0.05
gzip -8 180     0.08
gzip -9 180     0.12
xz -0   152     0.05
xz -1   148     0.06
xz -2   148     0.07
xz -3   148     0.08
xz -4   140     0.19
xz -5   136     0.23
xz -6   136     0.26
xz -7   136     0.27
xz -8   136     0.27
xz -9   136     0.27
bro -0  176     0.01
bro -1  176     0.01
bro -2  176     0.01
bro -3  172     0.01
bro -4  168     0.02
bro -5  156     0.03
bro -6  156     0.04
bro -7  156     0.04
bro -8  156     0.05
bro -9  156     0.07
bro -10 136     2.12
bro -11 136     2.10

gslapt decompression:

Comp.   Time
gzip -1 0.01
gzip -2 0.01
gzip -3 0.01
gzip -4 0.01
gzip -5 0.01
gzip -6 0.01
gzip -7 0.01
gzip -8 0.01
gzip -9 0.01
xz -0   0.02
xz -1   0.02
xz -2   0.02
xz -3   0.02
xz -4   0.02
xz -5   0.01
xz -6   0.02
xz -7   0.01
xz -8   0.01
xz -9   0.02
bro -0  0.00
bro -1  0.00
bro -2  0.00
bro -3  0.00
bro -4  0.00
bro -5  0.00
bro -6  0.00
bro -7  0.00
bro -8  0.00
bro -9  0.00
bro -10 0.00
bro -11 0.00

brotli compression

Comp.   Size    Time
gzip -1 924     0.06
gzip -2 904     0.06
gzip -3 880     0.08
gzip -4 868     0.09
gzip -5 852     0.12
gzip -6 852     0.16
gzip -7 848     0.17
gzip -8 848     0.28
gzip -9 848     0.54
xz -0   648     0.20
xz -1   332     0.15
xz -2   328     0.16
xz -3   328     0.17
xz -4   312     0.48
xz -5   296     0.59
xz -6   312     0.68
xz -7   312     0.68
xz -8   312     0.68
xz -9   312     0.68
bro -0  420     0.02
bro -1  420     0.02
bro -2  420     0.02
bro -3  424     0.02
bro -4  392     0.03
bro -5  380     0.09
bro -6  364     0.11
bro -7  360     0.13
bro -8  360     0.15
bro -9  356     0.18
bro -10 300     3.93
bro -11 300     3.94

brotli decompression:

Comp.   Time
gzip -1 0.03
gzip -2 0.03
gzip -3 0.03
gzip -4 0.03
gzip -5 0.03
gzip -6 0.03
gzip -7 0.03
gzip -8 0.03
gzip -9 0.03
xz -0   0.07
xz -1   0.04
xz -2   0.04
xz -3   0.04
xz -4   0.04
xz -5   0.04
xz -6   0.04
xz -7   0.04
xz -8   0.04
xz -9   0.04
bro -0  0.00
bro -1  0.00
bro -2  0.00
bro -3  0.00
bro -4  0.00
bro -5  0.00
bro -6  0.00
bro -7  0.00
bro -8  0.00
bro -9  0.00
bro -10 0.01
bro -11 0.01

firefox compression:

Comp.   Size    Time
gzip -1 49520   3.09
gzip -2 48904   3.24
gzip -3 48344   3.80
gzip -4 47684   3.99
gzip -5 47196   5.05
gzip -6 46952   6.79
gzip -7 46904   8.10
gzip -8 46828   12.74
gzip -9 46808   18.21
xz -0   43292   12.37
xz -1   42284   18.10
xz -2   41924   22.35
xz -3   41748   27.63
xz -4   40340   40.09
xz -5   39720   48.59
xz -6   39636   53.29
xz -7   39536   54.15
xz -8   39488   54.45
xz -9   39468   56.22
bro -0  47448   1.28
bro -1  47448   1.28
bro -2  47352   1.54
bro -3  47296   1.72
bro -4  46672   2.31
bro -5  43716   5.62
bro -6  43392   8.65
bro -7  42996   16.28
bro -8  42860   23.10
bro -9  42824   31.24
bro -10 40436   513.97
bro -11 40436   526.46

firefox decompression:

Comp.   Time
gzip -1 1.21
gzip -2 1.18
gzip -3 1.17
gzip -4 1.16
gzip -5 1.16
gzip -6 1.14
gzip -7 1.14
gzip -8 1.13
gzip -9 1.13
xz -0   3.36
xz -1   3.29
xz -2   3.26
xz -3   3.25
xz -4   3.32
xz -5   3.28
xz -6   3.30
xz -7   3.30
xz -8   3.34
xz -9   3.29
bro -0  0.55
bro -1  0.55
bro -2  0.54
bro -3  0.54
bro -4  0.55
bro -5  0.55
bro -6  0.55
bro -7  0.55
bro -8  0.55
bro -9  0.54
bro -10 0.83
bro -11 0.93

So, at least for these packages, it seems that brotli is not able to reach as small package sizes as xz, even in its higher settings. But compression and decompression times are very interesting, as both are a lot faster than even gzip, with better compression ratios in all cases. From these tests, I'd say it makes brotli a very strong contender for replacing gzip.

Very interesting, though one thing.

Originally Posted by gapan This got me interested so I did some tests. I compared all different quality settings for gzip, xz (default versions in slackware 14.1) and brotli (version 0.2.0).

Actually, Brotli goes to 11! No really, it does. It's top quality setting is 11.

P.S. 11 is very slow!

Using a crappy machine I had to hand I did a quick test compressing Firefox, with maximum settings. Now you see how slow Brotli 11 is at compression. However, it is close to XZ in size and much quicker on decompression.

$ wget -q http://ftp.slackware.no/slackware/slackware64-14.1/patches/packages/mozilla-firefox-38.3.0esr-x86_64-1_slack14.1.txz
$ xz -d mozilla-firefox-38.3.0esr-x86_64-1_slack14.1.txz
$ time xz -k9 mozilla-firefox-38.3.0esr-x86_64-1_slack14.1.tar 

real	0m56.512s
user	0m56.293s
sys	0m0.258s
$ time bro --quality 11 --input mozilla-firefox-38.3.0esr-x86_64-1_slack14.1.tar --output mozilla-firefox-38.3.0esr-x86_64-1_slack14.1.tar.br

real	8m2.319s
user	7m56.916s
sys	0m5.549s
$ ls -lh mozilla-firefox-38.3.0esr-x86_64-1_slack14.1.tar*
-rw-r--r-- 1 ruario users 101M Sep 22 21:05 mozilla-firefox-38.3.0esr-x86_64-1_slack14.1.tar
-rw------- 1 ruario users  42M Oct 20 20:16 mozilla-firefox-38.3.0esr-x86_64-1_slack14.1.tar.br
-rw-r--r-- 1 ruario users  41M Sep 22 21:05 mozilla-firefox-38.3.0esr-x86_64-1_slack14.1.tar.xz
$ time xz -cd mozilla-firefox-38.3.0esr-x86_64-1_slack14.1.tar.xz > /dev/null 

real	0m2.581s
user	0m2.552s
sys	0m0.027s
$ time bro --decompress --input mozilla-firefox-38.3.0esr-x86_64-1_slack14.1.tar.br > /dev/null 

real	0m0.676s
user	0m0.660s
sys	0m0.016s

Thought I would do another quick test. This time with a large package containing mainly text:

$ wget -q http://ftp.slackware.no/slackware/slackware64-14.1/slackware64/k/kernel-source-3.10.17-noarch-3.txz
$ xz -d kernel-source-3.10.17-noarch-3.txz 
$ time xz -k9 kernel-source-3.10.17-noarch-3.tar 

real	5m37.760s
user	5m36.777s
sys	0m1.218s
$ time bro --quality 11 --input kernel-source-3.10.17-noarch-3.tar --output kernel-source-3.10.17-noarch-3.tar.br

real	29m21.763s
user	29m6.457s
sys	0m16.497s
$ time gzip -k9 kernel-source-3.10.17-noarch-3.tar

real	0m34.046s
user	0m33.910s
sys	0m0.159s
$ ls -lh kernel-source-3.10.17-noarch-3.tar*
-rw-r--r-- 1 ruario users 503M Oct 23  2013 kernel-source-3.10.17-noarch-3.tar
-rw------- 1 ruario users  76M Oct 20 21:10 kernel-source-3.10.17-noarch-3.tar.br
-rw-r--r-- 1 ruario users 109M Oct 23  2013 kernel-source-3.10.17-noarch-3.tar.gz
-rw-r--r-- 1 ruario users  71M Oct 23  2013 kernel-source-3.10.17-noarch-3.tar.xz
$ time xz -cd kernel-source-3.10.17-noarch-3.tar.xz > /dev/null 

real	0m5.709s
user	0m5.635s
sys	0m0.077s
$ time bro --decompress --input kernel-source-3.10.17-noarch-3.tar.br > /dev/null 

real	0m1.318s
user	0m1.281s
sys	0m0.038s
$ time gzip  -cd kernel-source-3.10.17-noarch-3.tar.gz > /dev/null

real	0m3.175s
user	0m3.152s
sys	0m0.025s

Once again, Brotli could not quite match XZ for compression size and compression time is really long but decompression time is really good!

Originally Posted by ruario Actually, Brotli goes to 11! No really, it does. It's top quality setting is 11.

Hahah! I didn't know that! I just assumed it goes up to 9 as everything else does. Going up to 11 at least makes it louder than everything else!

Anyway, I have added quality settings 10 and 11 in my previous results. They both look very similar. At the same time they are very different from lower settings. Encoding times are indeed a lot slower, but sizes now are at the same level as xz -9, sometimes a bit smaller, sometimes a bit larger. But decompression times are a lot faster still.

OK, so I got really interested. I added (experimental) support for installing/upgrading .tbr packages in spkg. x86_64 package only for now here:
http://people.salixos.org/gapan/spkg/

Code is here: https://github.com/gapan/spkg

Install/upgrade .tbr packages with:

# spkg package.tbr

(you need to have brotli installed as well)

There's still the matter of creating the .tbr packages in the first place of course. For now, you can convert existing .txz packages with a combination of:

xzdec package.txz > package.tar
bro --input package.tar --output package.tbr

George, did you have a chance to try usm?

I guess it wouldn't be too complicated to just replace calls to {install,remove}pkg in it by calls to spkg instead? Just saying...

Sorry for the off-topic.