LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware
User Name
Password
Slackware This Forum is for the discussion of Slackware Linux.

Notices



Reply
 
Search this Thread
Old 01-27-2011, 02:59 PM   #1
Andy Alkaline
Member
 
Registered: Jun 2004
Location: Minnesota, USA
Distribution: Slackware32-stable, Debian-wheezy-amd64, LFS 7.1
Posts: 355

Rep: Reputation: 33
using chattr +c on ext4 fs


From what I've read, after I use chattr -R +c <dir>
the c flag is added and compression begins. And any file written to that directory afterward will automatically have the c flag added.

Also, next to the entry, it will show what type of compression was used (gzip,lzo, etc).

I've mounted my /home partition with the option user_xattr,

I've tried chattr on a few directories, and see the c flag has been added. It isn't added to new files written to the directory though, and it shows no compression method on any file when I use lsattr.

And using other utilities, I don't see any indication that compression took place.

I'm on Slackware 13.1 using vmlinuz-huge-smp-2.6.33.4-smp

Here is some output:
Quote:
andy@pigeon:~$ lsattr -a ~/.cache/google-chrome/*
--------c-I--e- /home/andy/.cache/google-chrome/Cache/.
--------c----e- /home/andy/.cache/google-chrome/Cache/..
--------c----e- /home/andy/.cache/google-chrome/Cache/f_00000b
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000016
--------c----e- /home/andy/.cache/google-chrome/Cache/f_00000e
--------c----e- /home/andy/.cache/google-chrome/Cache/f_00000c
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000006
--------c----e- /home/andy/.cache/google-chrome/Cache/f_00001d
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000015
--------c----e- /home/andy/.cache/google-chrome/Cache/data_0
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000003
--------c----e- /home/andy/.cache/google-chrome/Cache/f_00001a
--------c----e- /home/andy/.cache/google-chrome/Cache/data_2
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000009
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000014
--------c----e- /home/andy/.cache/google-chrome/Cache/index
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000010
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000021
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000004
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000007
--------c----e- /home/andy/.cache/google-chrome/Cache/data_1
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000005
--------c----e- /home/andy/.cache/google-chrome/Cache/f_00000a
--------c----e- /home/andy/.cache/google-chrome/Cache/f_00001f
--------c----e- /home/andy/.cache/google-chrome/Cache/f_00001e
--------c----e- /home/andy/.cache/google-chrome/Cache/f_00000f
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000011
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000013
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000017
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000008
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000024
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000022
--------c----e- /home/andy/.cache/google-chrome/Cache/f_00001b
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000023
--------c----e- /home/andy/.cache/google-chrome/Cache/f_00000d
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000025
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000012
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000018
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000002
--------c----e- /home/andy/.cache/google-chrome/Cache/data_3
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000001
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000019
--------c----e- /home/andy/.cache/google-chrome/Cache/f_000020
--------c----e- /home/andy/.cache/google-chrome/Cache/f_00001c
--------c----e- /home/andy/.cache/google-chrome/Media Cache/data_0
--------c----e- /home/andy/.cache/google-chrome/Media Cache/data_2
--------c----e- /home/andy/.cache/google-chrome/Media Cache/index
--------c----e- /home/andy/.cache/google-chrome/Media Cache/.
--------c----e- /home/andy/.cache/google-chrome/Media Cache/data_1
--------c----e- /home/andy/.cache/google-chrome/Media Cache/data_3
--------c----e- /home/andy/.cache/google-chrome/Media Cache/..
 
Old 01-28-2011, 08:32 AM   #2
gnashley
Amigo developer
 
Registered: Dec 2003
Location: Germany
Distribution: Slackware
Posts: 4,776

Rep: Reputation: 481Reputation: 481Reputation: 481Reputation: 481Reputation: 481
Unless I'm mistaken, the compression is transparent (inline), so you won't notice anything about the files when reading them or viewing their properties -unless you use lsattr. You might also make copies of a file or two and place them in the dir you use 'attr -c' on. Then compare the (hopefully compressed) files using du with and without the '--apparent-size' option.
 
1 members found this post helpful.
Old 01-28-2011, 01:31 PM   #3
Andy Alkaline
Member
 
Registered: Jun 2004
Location: Minnesota, USA
Distribution: Slackware32-stable, Debian-wheezy-amd64, LFS 7.1
Posts: 355

Original Poster
Rep: Reputation: 33
Quote:
andy@pigeon:~$ cp -R .cache/google-chrome/ temp
andy@pigeon:~$ du -s .cache/google-chrome/ | tee log ; du -s temp | tee -a log
2256 .cache/google-chrome/
2408 temp
Checking the properties using pcmanfm
Total size of Files:
7.6 .cache/google-chrome/
7.7 temp

Size on Disk:
18.5 .cache/google-chrome/
19.7 temp

Well now, this is interesting....

Using rsync instead of cp to mirror the directory

Quote:
ndy@pigeon:~$ rsync -avh --delete .cache/google-chrome/ temp
sending incremental file list
./
Cache/
Cache/data_0
Cache/data_1
Cache/data_2
Cache/data_3
Cache/f_000001
Cache/f_000002
Cache/f_000003
Cache/f_000004
Cache/f_000005
Cache/f_000006
Cache/f_000007
Cache/f_000008
Cache/f_000009
Cache/f_00000a
Cache/f_00000b
Cache/f_00000c
Cache/f_00000d
Cache/f_00000e
Cache/f_00000f
Cache/f_000010
Cache/f_000011
Cache/f_000012
Cache/index
Media Cache/
Media Cache/data_0
Media Cache/data_1
Media Cache/data_2
Media Cache/data_3
Media Cache/index

sent 7.88M bytes received 555 bytes 15.76M bytes/sec
total size is 7.88M speedup is 1.00
andy@pigeon:~$ du -sh .cache/google-chrome/ | tee -a log ; du -sh temp | tee -a log
2.3M .cache/google-chrome/
7.6M temp
andy@pigeon:~$ du -sh .cache/google-chrome/ | tee -a log ; du -sh temp | tee -a log
2.3M .cache/google-chrome/
7.6M temp
After deleting temp and using cp again:
Quote:
andy@pigeon:~$ cp -R .cache/google-chrome/ temp
andy@pigeon:~$ du -sh .cache/google-chrome/ ; du -sh temp
2.3M .cache/google-chrome/
2.4M temp
Radically different results using rsync; why might that be?

So, the compression appears to be working (though in this case I'm not gaining much). Thanks, gnashley.
 
Old 01-28-2011, 03:23 PM   #4
gnashley
Amigo developer
 
Registered: Dec 2003
Location: Germany
Distribution: Slackware
Posts: 4,776

Rep: Reputation: 481Reputation: 481Reputation: 481Reputation: 481Reputation: 481
That looks like over 50% savings which is quite good! Saviings with a buch of small files may not be as good as with larger files.

About rsync and pcmanfm, you have to realize that there is more than one way to determine the size of a file and most tools -especially file managers- will only use one method. To get the best idea of the savings, simply compare one or two representative files by using 'du' -first normally as you usually would. Then run it but add '--apparent-size' to the options. I have for a long time used (occassionaly) an ext2 modification called e2compr -it's a patch for the kernel, plus patched tools like findutils and e2fsprogs. The kernel patch makes the kernel recognize and transparently compress or decompress the files. The findutils and e2fsprogs patches make it possible to change the attributes of files and dirs and to search for such compressed files. The e2compr code never made it into the official ext2/ext3 code, but may have served as a base or idea for the current ext4 support for compression. I think it'S great because you just 'chattr' a directory and then everything you put in there gets compressed without any other steps.

I'm not sure whether the ext4 code is working the same way or not. with the old code, you had to use chattr *first* and then anything written there would be compressed -and you could undo the compression with chattr, if needed. Then files added would not be compressed -but the old ones would still remain compressed.

I found the ext2 support(old code) quite stable and gave results comparable with what you are seeing -in the range of 45-60% depending on the specific files. I still hope to see generic compression support (maybe at the VFS level) which would provide compression for any supported file system. If I understand correctly, btrfs is going to have or does have compression support.

About mirroring the directory contents -unless you use chattr to set the compression active for the new directory, the files should appear uncompressed in the new dir. Since the kernel decompresses them as they are read, tools like cp and rsync are receiving them already uncompressed.

Last edited by gnashley; 01-28-2011 at 03:26 PM.
 
1 members found this post helpful.
Old 01-28-2011, 07:08 PM   #5
Andy Alkaline
Member
 
Registered: Jun 2004
Location: Minnesota, USA
Distribution: Slackware32-stable, Debian-wheezy-amd64, LFS 7.1
Posts: 355

Original Poster
Rep: Reputation: 33
I didn't follow your instructions well the first time. I also didn't know about the --apparent-size option of du. Thanks for the lessons.

andy@pigeon:~$ du -sh --apparent-size ~/.cache/google-chrome/
24M /home/andy/.cache/google-chrome/
andy@pigeon:~$ du -sh ~/.cache/google-chrome/
20M /home/andy/.cache/google-chrome/
 
Old 01-28-2011, 09:16 PM   #6
Andy Alkaline
Member
 
Registered: Jun 2004
Location: Minnesota, USA
Distribution: Slackware32-stable, Debian-wheezy-amd64, LFS 7.1
Posts: 355

Original Poster
Rep: Reputation: 33
Just some extra info; compared it against two uncompressed directories.

Quote:
andy@pigeon:~$ du -sh ~/.cache/google-chrome/
23M /home/andy/.cache/google-chrome/
andy@pigeon:~$ du -sh --apparent-size ~/.cache/google-chrome/
27M /home/andy/.cache/google-chrome/

andy@pigeon:~$ du -sh /opt
783M /opt
andy@pigeon:~$ du -sh --apparent-size /opt
753M /opt

ndy@pigeon:~$ du -sh /usr/local/
161M /usr/local/
andy@pigeon:~$ du -sh --apparent-size /usr/local/
158M /usr/local/
From the du man page
Quote:
--apparent-size
print apparent sizes, rather than disk usage; although the
apparent size is usually smaller, it may be larger due to holes
in (`sparse') files, internal fragmentation, indirect blocks,
and the like
 
Old 01-29-2011, 02:57 AM   #7
gnashley
Amigo developer
 
Registered: Dec 2003
Location: Germany
Distribution: Slackware
Posts: 4,776

Rep: Reputation: 481Reputation: 481Reputation: 481Reputation: 481Reputation: 481
Well, that doesn't seem to be much compression after all. Have you been able to find out what type of compression is being used? The old patches I worked with allowed for lzo, gzip or bzip2.
 
Old 01-29-2011, 06:21 PM   #8
Andy Alkaline
Member
 
Registered: Jun 2004
Location: Minnesota, USA
Distribution: Slackware32-stable, Debian-wheezy-amd64, LFS 7.1
Posts: 355

Original Poster
Rep: Reputation: 33
Most of the examples I've found were people using E2compr, and in the quote below, it illustrates that the compression used is shown next to the file name when using lsattr. But I don't have anything like that showing when I use lsattr myself. Specific to ext2? Outdated info? I really couldn't find much info in google using compression with an ext4 fs, and searches on the subject chattr or "chattr compress" brings up mostly manual pages.

http://linuxgazette.net/18/e2compr.html
Quote:
E2compr is almost too transparent. After rebooting the patched kernel of course the first thing I wanted to do was to compress some nonessential files and see what would happen. Using the modified chattr command, chattr +c * will set the new compression flag on every file in the current directory. Oddly enough, though, running ls -l on the directory afterwards shows the same file sizes! I found that the only way to tell how much disk space has been saved is to run du on the directory both before and after the compression attribute has been toggled. Evidently du and ls use different methods of determining sizes of files. If you just want to see if a file or directory has been compressed, running the patched lsattr on it will result in something like this:


%-> lsattr libso312.so
--c---- 32 gzip9 libso312.so
The "c" in the third field shows that the file is compressed, "gzip9" is the compression algorithm used, and "32" is the blocksize. If a file hasn't been compressed the output will just be a row of dashes.
 
Old 01-30-2011, 04:20 AM   #9
gnashley
Amigo developer
 
Registered: Dec 2003
Location: Germany
Distribution: Slackware
Posts: 4,776

Rep: Reputation: 481Reputation: 481Reputation: 481Reputation: 481Reputation: 481
The linuxgazette article is really old, but i nteresting beecause I had never run across it. e2compr is what I mentioned that I have worked with. There have been no updates now to the sourceforge site for wuite a while, but the some of the last work was done on porting e2compr to e3compr -for ext3 filesystems. Matthias Winkler is the current developer -I've had a bit of email contact with him before. For e3compr, he had trimmed the code considerably -taking out several of the compression types, which I thought made good sense. He talked to me about trying to implement a general VFS-level compressor -but it would be really hard to get such code into the mainstream kernel.

I'd like to see the current code modified to compile and run as en external kernel module -that would keep one from having to constatntly update the kernel patches to match new kernel versions. The e3compr version of the code may still be unusable -I need to get in contatc with Matthias again and see what the status is. Still, I am glad to see compression implemented in more than one of the filesystems available in the standard kernel.
 
  


Reply

Tags
attributes, compression, du, ext4


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Is it safe to format USB flash to ext4 or ext4? joham34 Linux - Newbie 2 01-08-2011 12:58 PM
chattr not working JJX Linux - Newbie 3 10-18-2005 04:39 AM
chattr + 2.4.20-lq2? tarballedtux Linux - Security 3 01-06-2003 09:30 AM
need help with chattr ruprick Linux - Newbie 4 02-19-2002 11:39 AM
chattr usage sancho5 Linux - General 0 11-21-2001 03:02 PM


All times are GMT -5. The time now is 07:29 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration