Economy of data storage

w1k0 · 01-01-2009, 01:21 PM

Until now I used disks divided into eight partitions: two for systems, one for swap, and four for data storage (as well as one extended partition). Each partition was dedicated for some kind of data. When I saw that one of them is on the verge of fill up I removed some superfluous data or moved selected kind of data from full partition to empty one. It was convenient and logical.

Now I stated I waste a lot of disk space. Some data grow slow and some grow fast. On one partition I have 50% of free space and on the other merely 5%. I need to organize anew data on occupied partition while I have a lot of space on another one. So I decided to put all data to one big partition.

I use two machines: first and second. The first has 160 GB SATA HDD and the second -- 80 GB ATA HDD. On the second machine I keep backups of some valid data from the first machine. It's impossible to backup it entirely because the second hard drive is two times smaller then the first. Until now both these machines used the same pattern of partitions.

Now I decided to test new type of partitioning on the second machine.

This is the first machine with original eight partitions:

$ fdisk -l /dev/sda

Code:

Disk /dev/sda: 160.0 GB, 160041885696 bytes
240 heads, 63 sectors/track, 20673 cylinders
Units = cylinders of 15120 * 512 = 7741440 bytes
Disk identifier: 0xcccdcccd

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        1000     7559968+  83  Linux
/dev/sda2            1001        2000     7560000   83  Linux
/dev/sda3            2001        6600    34776000   83  Linux
/dev/sda4            6601       20673   106391880    5  Extended
/dev/sda5            6601       11200    34775968+  83  Linux
/dev/sda6           11201       15800    34775968+  83  Linux
/dev/sda7           15801       20400    34775968+  83  Linux
/dev/sda8           20401       20673     2063848+  82  Linux swap

This is the second machine after replacing partitions from 3 to 7 by one huge partition 3:

# fdisk -l /dev/hda

Code:

Disk /dev/hda: 80.0 GB, 80026361856 bytes
240 heads, 63 sectors/track, 10337 cylinders
Units = cylinders of 15120 * 512 = 7741440 bytes
Disk identifier: 0x3db012b3

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1               1        1025     7748968+   c  W95 FAT32 (LBA)
/dev/hda2            1026        2050     7749000   83  Linux
/dev/hda3            2051       10250    61992000   83  Linux
/dev/hda4           10251       10337      657720   82  Linux swap

On all Linux partitions I use Reiser File System.

After repartitioning the disk I copied data from partitions 3 and 7 on the first machine to partition 3 on the second machine.

This is the first machine with original data on partitions 3 and 7:

$ df

Code:

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1              7559700   5218112   2341588  70% /
tmpfs                   516672         0    516672   0% /dev/shm
/dev/sda3             34774932   7114156  27660776  21% /mnt/sda3
/dev/sda5             34774868  17603104  17171764  51% /mnt/sda5
/dev/sda6             34774868  10900692  23874176  32% /mnt/sda6
/dev/sda7             34774868  10739128  24035740  31% /mnt/sda7

This is the second machine and new data on partition 3:

# df

Code:

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda2              7748748   6895612    853136  89% /
tmpfs                   257260         0    257260   0% /dev/shm
/dev/hda1              6790684   4091536   2699148  61% /mnt/hda1
/dev/hda3             61990104  19293484  42696620  32% /mnt/hda3

Partitions 3 and 7 on the first machine use 7114156 + 10739128 = 17853284 1K-blocks of disk space. Partition 3 on the second machine stores exactly the same data and uses 19293484 1K-blocks of disk space. The same data occupies on the second hard drive 8% more disk space than on the first hard drive.

I wonder what makes this difference. Are smaller partitions more economic than big ones? Does it depend on the used file system? Does it depend on hard drive geometry? Is there another reason of such behavior of data on these disks?

Any help will be welcomed.

Jeebizz · 01-01-2009, 02:16 PM

I am going to guess and possibly agree with you on the size of the partition and it's efficiency. Particularly if you use and ext2/3 partition and it is 100+GB, 15%+ of that is reserved for system. Normally that doesn't seem much, but when I noticed 30GB of my 200GB drive being reserved on an ext3 partition, I quickly decided not to use that FS. I don't really know how Reiser handles things. Well, sorta. I remember I did format a 200GB partition into ReiserFS and it took a very long time just to mount. I think we as users should take a look at tweaking said filesystems if we do decide to use a gianormous partition instead of a smaller one.

In your case you might want to check the man pages for resierfstune.

Code:

man reiserfstune

salasi · 01-01-2009, 02:28 PM

Reiser is probably a good choice for partitions with lots of small files; don't remember the comparison (ie, what exactly it was compared with) but I remember a lower storage space loss.

Note that if you get too clever with partitons, it does inevitably mean loss of space.

You only really have to back up /home and key config files (usually, /etc will do it, but maybe there is some odd stuff under /root), provided that you still do a clean install. Maybe you are backing up stuff that you can get from your install disk.

alkos333 · 01-01-2009, 03:46 PM

Why not use LVM, so that you can resize your partitions on the go as you need to? With LVM you can also take snapshots of your partitions.

If you can, do a full back-up because it will be easier restore.

w1k0 · 01-01-2009, 06:07 PM

Jeebizz, reiserfstune help to change journal size, transaction size, and journal location. I'll read more about it to state if it's possible to tune ReiserFS for significantly better work but I suppose these changes have rather cosmetic character though on large volumes effects can be probably distinct.

salasi, you're right when you stated: ``Note that if you get too clever with partitons, it does inevitably mean loss of space''. A few years ago I used a total of 15 partitions. Then I used a total of 8 partitions. Now I consider to use a total of 4 partitions.

alkos333, LVM2 is perhaps the best solution if you'd like to change volumes sizes on the fly. I appreciate that great mechanism and I'll read more about it but it seems to me that I'm too conservative to use it. I fear it's easy to loose all data in case of some crash.

Thank you all guys for your replies.

w1k0 · 01-02-2009, 10:58 AM

I just reorganized data on the first machine switching from a few smaller partitions to one big one:

$ fdisk -l /dev/sda

Code:

Disk /dev/sda: 160.0 GB, 160041885696 bytes
240 heads, 63 sectors/track, 20673 cylinders
Units = cylinders of 15120 * 512 = 7741440 bytes
Disk identifier: 0xcccdcccd

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        1000     7559968+  83  Linux
/dev/sda2            1001        2000     7560000   83  Linux
/dev/sda3            2001       20400   139104000   83  Linux
/dev/sda4           20401       20673     2063880   82  Linux swap

$ df

Code:

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1              7559700   5099532   2460168  68% /
tmpfs                   516672         0    516672   0% /dev/shm
/dev/sda3            139099748  19293460 119806288  14% /mnt/sda3

The same set of data occupies:

1. 17853284 1K-blocks of disk space on two partitions each of 34.77 GB on 160 GB SATA HDD (100%)

2. 19293484 1K-blocks of disk space on one partition of 61.99 GB on 80 GB ATA HDD (108%)

3. 19293460 1K-blocks of disk space on one partition of 139.09 GB on 160 GB SATA HDD (108%)

In the second and the third cases journaling system occupied 1% of disk space on empty partition. I didn't measure that in the first case.

Surprisingly there was no any disk space loss when I switched from 60 GB partition to 140 GB partition though there was disk space loss when I switched from two smaller partitions of 35 GB to one bigger of 60 GB. It seems that there is more economical to use a few partitions than one but there is no difference between small and big partitions.

I measured the time needed to check the file systems. In the case of four partitions each of 35 GB it was about 15 sec. in total -- in the case of one partition of 140 GB it was about 10 sec.

gargamel · 01-03-2009, 04:17 AM

Quote:

Originally Posted by salasi

Reiser is probably a good choice for partitions with lots of small files; don't remember the comparison (ie, what exactly it was compared with) but I remember a lower storage space loss.

Filesystems like ext3 and ReiserFS are usually good for "small" files up to 100 MB per file. For bigger files XFS or JFS are usually recommended alternatives.
In the last two comparisons I saw JFS came out to be the best allround filesystem for Linux, because it shows good performance in all typical scenarios. And it has now good support in Slackware 12.2.

In the past I used ReiserFS, and never had problems with it. Ext3 on the other hand, did cause me a lot of problems, because it failed to handle large files reliably. Since Slackware 12.1 these troubles are gone, and I use it, because it is "the default".

For read-only partitions I use ext2, because there the overhead of journaling just makes no sense.

Quote:

Originally Posted by salasi

Note that if you get too clever with partitons, it does inevitably mean loss of space.

YES!!!

Quote:

Originally Posted by salasi

You only really have to back up /home and key config files (usually, /etc will do it, but maybe there is some odd stuff under /root), provided that you still do a clean install. Maybe you are backing up stuff that you can get from your install disk.

If you are running a database or webserver some other directories may be worth to be considered, also, such as /srv.

gargamel