LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel
User Name
Password
Linux - Kernel This forum is for all discussion relating to the Linux kernel.

Notices


Reply
  Search this Thread
Old 09-21-2016, 03:57 AM   #16
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,849

Rep: Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309

zram will back up files if there was enough space to do that. Otherwise it will keep only the compressed data and will uncompress files on demand.
In your case you can check the compression ratio by manually compressing your binaries. As far as I see you will not be able to save huge amount of ram using zram.
 
Old 09-21-2016, 04:25 AM   #17
tsenthilnath
LQ Newbie
 
Registered: Aug 2016
Posts: 12

Original Poster
Rep: Reputation: Disabled
"zram will back up files if there was enough space to do that. Otherwise it will keep only the compressed data and will uncompress files on demand. " - This is a news to me !

Neither the Linux documentation (https://www.kernel.org/doc/Documenta...ckdev/zram.txt) talks about it nor I don't see this info in the internet as well. Can you please point out to me where this is documented/discussed ?

If this is true, can you suggest an alternate way to save store and retrieve binaries like tmpfs , but can save memory ?

Thanks,
Senthil
 
Old 09-21-2016, 04:26 AM   #18
tsenthilnath
LQ Newbie
 
Registered: Aug 2016
Posts: 12

Original Poster
Rep: Reputation: Disabled
"zram will back up files if there was enough space to do that. Otherwise it will keep only the compressed data and will uncompress files on demand. " - This is a news to me !

Neither the Linux documentation (https://www.kernel.org/doc/Documenta...ckdev/zram.txt) talks about it nor I don't see this info in the internet as well. Can you please point out to me where this is documented/discussed ?

If this is true, can you suggest an alternate way to save store and retrieve binaries like tmpfs , but can save memory ?

Thanks,
Senthil
 
Old 09-21-2016, 04:45 AM   #19
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,849

Rep: Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309Reputation: 7309
http://www.robert-franz.com/2015/12/...ressed-memory/
probably here you can find a better explanation: http://superuser.com/questions/45098...che-swapcached, look for the long answer.
Unfortunately I cannot suggest any other solution. Probably using (more) shared objects may lower the required space, but we need to know much more about your configuration to give you any hint. Probably you can optimize your apps to use disk instead of ram...
 
Old 05-02-2017, 04:27 AM   #20
raymond18
LQ Newbie
 
Registered: May 2008
Posts: 6

Rep: Reputation: 0
Hi tsenthilnath,

Did you resolve the problem? Now I face the same problem when I try to use zram as block devices.
The zram block devices seem like normal tmpfs without any compress.
 
Old 05-02-2017, 01:34 PM   #21
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,659
Blog Entries: 4

Rep: Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941
Offhand, it sounds to me like you need to buy silicon. If "1.5GB" in a RAM-disk is causing you grief, you simply don't own enough physical RAM.

And if the motherboard won't take more RAM, well, "it's long-since been time to shoot retire that old nag, anyway ..."
 
Old 05-03-2017, 09:22 AM   #22
raymond18
LQ Newbie
 
Registered: May 2008
Posts: 6

Rep: Reputation: 0
Hi Sundialsvcs,
Actually, it's hard for me to add more memory into motherboard, because I try to run my customize OS on embedded systems currently.
All I want is use the limited memory(1GB) as effective as I can.
I use the following commands to create a zram devices and format it as ext4 fs then mount it.
After that I use dd to create many dummy files, however, it seems not compress at all.

PHP Code:
root@host $ /sys/block sudo zramctl ---size 1G
/dev/zram0

root
@host sudo mkfs.ext4 /dev/zram0
mke2fs 1.42.13 
(17-May-2015)
Discarding device blocksdone
Creating filesystem with 262144 4k blocks 
and 65536 inodes
Filesystem UUID
e1d3dd51-931f-4a67-9e5c-816c0834f370
Superblock backups stored on blocks
:
    
3276898304163840229376
Allocating group tables
done
Writing inode tables
done
Creating journal 
(8192 blocks): done
Writing superblocks 
and filesystem accounting informationdone

root
@host sudo mount /dev/zram0 /mnt/zram/

root@host sudo dd if=/dev/zero of=/mnt/zram/test bs=4M count=60
root
@host sudo dd if=/dev/zero of=/mnt/zram/test2 bs=4M count=50
root
@host sudo dd if=/dev/zero of=/mnt/zram/test3 bs=4M count=60
root
@host sudo dd if=/dev/zero of=/mnt/zram/test4 bs=4M count=60
dd
error writing '/mnt/zram/test4'No space left on device
55
+0 records in
54
+0 records out
230621184 bytes 
(231 MB220 MiBcopied0.157522 s1.5 GB/s

root
@host df -h|grep zram
/dev/zram0      1.0G  917M  224K 100% /mnt/zram

root
@host ls -lh
total 900M
-rw-r--r--. 1 root root 240M May  2 07:23 test
-rw-r--r--. 1 root root 200M May  2 07:20 test2
-rw-r--r--. 1 root root 240M May  2 07:22 test3
-rw-r--r--. 1 root root 220M May  2 07:24 test4
root
@host zramctl 
NAME       ALGORITHM DISKSIZE  DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 lzo             1G  4.2M  180K  568K       4 
Does zram disk only be compressed when it works as swap disk?
Any idea plz?

Last edited by raymond18; 05-04-2017 at 12:34 AM.
 
Old 05-03-2017, 02:14 PM   #23
IsaacKuo
Senior Member
 
Registered: Apr 2004
Location: Baton Rouge, Louisiana, USA
Distribution: Debian Stable
Posts: 2,546
Blog Entries: 8

Rep: Reputation: 465Reputation: 465Reputation: 465Reputation: 465Reputation: 465
I don't see anything in the post showing the commands to create the zram device, but it looks like the size of the ext4 file system is 1.0G in size. Considering the default settings and typical overhead, 917M of data is a reasonable limit on how much stuff could fit in there (the uncompressed size).

It looks like the compressed size is 568K, but I'm not positive. The output of zramctl is not immediately obvious in meaning.

If you want to store more than 1GB in an ext4 file system, you must make the zram block device larger than 1GB (the uncompressed size).

I stopped experimenting with using ext4 on zram when it became obvious that it was hideously inefficient for my uses. It's okay if you mostly just write once, or the total disk usage is consistently near the maximum. But otherwise, you end up with a lot of dirty blocks which are merely deallocated rather than wiped clean. Zram has no way to know thse blocks can be safely discarded, so they continue to consume ram.

My preference is to just use tmpfs, because it is aware when data can be discarded. Of course, it's not compressed, but I like how the ram usage doesn't grow and grow.

OTOH, my issue could be solved by setting up something like zerofree to run periodically - zeroing out freed blocks.
 
1 members found this post helpful.
Old 05-04-2017, 12:40 AM   #24
raymond18
LQ Newbie
 
Registered: May 2008
Posts: 6

Rep: Reputation: 0
Hi IsaacKuo,
Thanks for ur reply. I add the commands above how I create, format and mount zram device.

It's ok for me to use any other file system rather than ext4 if another one can satisfy what I need (store more data/files than 1GB through compressing them).

Or is there any way to achieve this except zram?

Last edited by raymond18; 05-04-2017 at 12:42 AM.
 
Old 05-04-2017, 09:27 AM   #25
IsaacKuo
Senior Member
 
Registered: Apr 2004
Location: Baton Rouge, Louisiana, USA
Distribution: Debian Stable
Posts: 2,546
Blog Entries: 8

Rep: Reputation: 465Reputation: 465Reputation: 465Reputation: 465Reputation: 465
Quote:
Originally Posted by raymond18 View Post
Hi IsaacKuo,
Thanks for ur reply. I add the commands above how I create, format and mount zram device.

It's ok for me to use any other file system rather than ext4 if another one can satisfy what I need (store more data/files than 1GB through compressing them).

Or is there any way to achieve this except zram?
ext4 is fine. Your main problem is the first command:

sudo zramctl -f --size 1G

This will create a block device with a size of 1G (uncompressed). It will never be able to store more than 1G of data. The file system overhead and default settings will limit you to some amount a bit less than that.

If you want to store more data than that, you must create the zram block device with a larger size. Note that the compressed size will still need to fit in ram, so there's not much point to making this size bigger than, say, twice the amount of physical ram. This depends on how well it compresses, of course. Depending on the exact nature of the data being stored, you might get higher compression rates. Or you might get hardly any compression. For example, if the bulk of your data is jpg files, then you're better off just using ordinary tmpfs (no compression).
 
Old 05-04-2017, 08:43 PM   #26
raymond18
LQ Newbie
 
Registered: May 2008
Posts: 6

Rep: Reputation: 0
As you say
> This depends on how well it compresses, of course.... you might get higher compression rates

I just want to clarify my understanding of zram block deivce. In my cognition, when I restore the files/data into zram device, the zram device will help me to compress(LZO OR LZ4) them at first and write into block device later.

For example, there are two file
PHP Code:
ls -lh |grep rootfs
-rw-r--r--.  1 root root  54M May  5 01:18 rootfs.tar
-rw-r--r--.  1 root root  21M May  5 01:17 rootfs.tar.gz
df -h|grep zram
Filesystem      Size  Used Avail 
Use% Mounted on
/dev/zram0      1.0G   17M  899M   2% /mnt/zram 
The rootfs.tar is the file I want to restore in zram and the rootfs.tar.gz is the compressed one by gzip.
If I cp the rootfs.tar to the normal tmpfs, the total size of course will be 17M(Used) + 54M(rootfs.tar) = 71M .
But if I cp the rootfs.tar to the zram device, I expect the size of "Used" should be much less then 71M, because zram will help me to compress rootfs.tar automatically.

However, the result is not my result, it seems not compress rootfs.tar at all.
Or what's the right method to use zram?

PHP Code:
sudo cp ./rootfs.tar /mnt/zram/
df -h|grep zram
/dev/zram0      1.0G   71M  846M   8% /mnt/zram 
 
Old 05-05-2017, 11:07 AM   #27
IsaacKuo
Senior Member
 
Registered: Apr 2004
Location: Baton Rouge, Louisiana, USA
Distribution: Debian Stable
Posts: 2,546
Blog Entries: 8

Rep: Reputation: 465Reputation: 465Reputation: 465Reputation: 465Reputation: 465
df will only report the uncompressed sizes, because it is telling you about what the ext4 filesystem layer sees. The ext4 filesystem knows nothing about the fact that the block device it's writing to/from is a zram device.

In order to determine how much ram is consumed by the zram block device, you need to use zramctl.
 
Old 05-05-2017, 11:09 AM   #28
IsaacKuo
Senior Member
 
Registered: Apr 2004
Location: Baton Rouge, Louisiana, USA
Distribution: Debian Stable
Posts: 2,546
Blog Entries: 8

Rep: Reputation: 465Reputation: 465Reputation: 465Reputation: 465Reputation: 465
You might find it less confusing if you create the zram device with a much bigger size, like this:

sudo zramctl -f --size 5G

This is massive overkill, but you might understand the numbers better because you know there is much less than 5G of physical RAM in the computer.
 
Old 05-06-2017, 08:19 AM   #29
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,659
Blog Entries: 4

Rep: Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941
I don't know ZRAM, but I ponder if it is trying to tell you the uncompressed data size . . . ?

Also, it might have to hold the data briefly in order to compress it, and perhaps it does not (seem to) release that space right away. Unless something is exerting pressure on the memory subsystem, it tends to be very "lazy."
 
Old 05-08-2017, 11:35 AM   #30
IsaacKuo
Senior Member
 
Registered: Apr 2004
Location: Baton Rouge, Louisiana, USA
Distribution: Debian Stable
Posts: 2,546
Blog Entries: 8

Rep: Reputation: 465Reputation: 465Reputation: 465Reputation: 465Reputation: 465
The df command knows nothing about zram, and will never report the compressed sizes. It is only reporting what the ext4 file system reports to it (the ext4 file system knows nothing about the fact that the block device it is using is compressed or is in ram).
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Where did zram go? Slax-Dude Slackware 5 10-31-2015 01:26 PM
Compression ratio and mplayer stdout output. stf92 General 0 07-06-2011 03:19 AM
Determining the compression ratio when using -9 with gzip kaplan71 Linux - Software 2 04-08-2010 03:45 PM
Slax with SquashFS-4 new compression algorithm and layered compression ratios? lincaptainhenryjbrown Linux - Software 2 06-19-2009 05:29 PM
File Roller: how to set compression ratio? How to preserve all color info in JPEGs? bezdomny Linux - Software 3 03-30-2009 12:53 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel

All times are GMT -5. The time now is 09:06 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration