Zone says "no space left on device" in /tmp when there's plently?

AlucardZero · 08-18-2010, 11:51 AM

I have an M3000 running a few zones. On the zone, when I try to extract a ~3GB file in /tmp, GNU tar runs for a while and says "No space left on device" (Sun tar says "HELP - extract write error"). But df says there's 8GB free in /tmp!

Host:

Code:

zonehost3:~$ uname -a
SunOS zonehost3 5.10 Generic_142900-14 sun4u sparc SUNW,SPARC-Enterprise
zonehost3:~$ zfs list
NAME                               USED  AVAIL  REFER  MOUNTPOINT
zonepool                          79.0G   313G    27K  /zonepool
zonepool/sol01                 9.20G  60.8G  9.19G  /zonepool/sol01
zonepool/swap                        8G   318G  3.27G  -
zonehost3:~$ swap -s
total: 20722336k bytes allocated + 40816k reserved = 20763152k used, 8551208k available
zonehost3:~$ swap -l
swapfile             dev  swaplo blocks   free
/dev/dsk/c0t0d0s1   32,9      16 8425712 3873104
/dev/dsk/c0t0d0s3   32,11     16 8425712 3898128
/dev/zvol/dsk/zonepool/swap 256,1      16 16777200 12219296
zonehost3:~$ zfs get volsize zonepool/swap
NAME           PROPERTY  VALUE    SOURCE
zonepool/swap  volsize   12G      -
zonehost3:~$ zonecfg -z sol01 info
zonename: sol01
zonepath: /zonepool/sol01
brand: native
autoboot: true
bootargs: 
pool: 
limitpriv: 
scheduling-class: FSS
ip-type: shared
net:
        address: 1.2.3.15/24
        physical: bge0
        defrouter: 1.2.3.1
capped-cpu:
        [ncpus: 2.00]
capped-memory:
        physical: 6G
        [swap: 12G]
rctl:
        name: zone.cpu-cap
        value: (priv=privileged,limit=200,action=deny)
rctl:
        name: zone.max-swap
        value: (priv=privileged,limit=12884901888,action=deny)

Zone:

Code:

sol01:/tmp$ df -h
Filesystem             size   used  avail capacity  Mounted on
/                        0K   9.2G    61G    14%    /
/dev                    70G   9.2G    61G    14%    /dev
proc                     0K     0K     0K     0%    /proc
ctfs                     0K     0K     0K     0%    /system/contract
mnttab                   0K     0K     0K     0%    /etc/mnttab
objfs                    0K     0K     0K     0%    /system/object
swap                   8.2G   336K   8.2G     1%    /etc/svc/volatile
fd                       0K     0K     0K     0%    /dev/fd
swap                    20G    12G   8.2G    60%    /tmp
swap                   8.2G    32K   8.2G     1%    /var/run
sol01:/tmp$ swap -l
swapfile             dev  swaplo blocks   free
/dev/swap           4294967295,4294967295     16 25165824 248096
sol01:/tmp$ swap -s
total: 12458848k bytes allocated + 0k reserved = 12458848k used, 124064k available
sol01:/tmp$ ls -l suite.100816.tar.gz 
-rw-r--r--   1 user    group 3714639805 Aug 18 07:49 suite.100816.tar.gz
sol01:/tmp$ gunzip < suite.100816.tar.gz | /usr/local/bin/tar xvf -
#[...]
/usr/local/bin/tar: suite/foor/libicudata.so.24.0: Cannot write: No space left on device

Why can't I extract the whole archive?

yooy · 08-18-2010, 02:26 PM

have you tried to extract to different location?
you can insert usb key and extract there

AlucardZero · 08-18-2010, 02:35 PM

Sure I can extract elsewhere, but how can I solve this so people don't get confused the same way in the future?

druuna · 08-18-2010, 02:53 PM

Hi,

A .tar.gz file is not an archive. Its a compressed archive!

Unless you know the size of the uncompressed archive, which can be many times bigger then the 3+Gb compressed archive, you cannot tell how big the uncompressed version will be.

Example:
$ ls -lh binutils-2.20.tar.gz
-rw-r----- 1 druuna internet 23M Oct 16 2009 binutils-2.20.tar.gz
$ gunzip binutils-2.20.tar.gz
$ ls -lh binutils-2.20.tar
-rw-r----- 1 druuna internet 123M Oct 16 2009 binutils-2.20.tar

Almost 6 times the size.

Another thing: When you gunzip a file you end up with only the .tar version. But during the process of decompressing both are present until the end. So you actually need 7 times the size in my example.

You have to find a partition/disk with more space to gunzip that file.....

Hope this clears things up a bit.

AlucardZero · 08-18-2010, 03:05 PM

it's about 4.5GB uncompressed and unpacked, so it will fit.

Also, I unpack with:

Code:

gunzip < filename.tar.gz | tar xvf -

which removes the need to have a .tar file ever present.

So what I think I'm running into is:
a) the host has ~19GB swap
b) the zone is limited to ~12G virtual address space
c) tmp is mounted on RAM like always
d) 'df' in a zone shows the host swap size for tmpfs filesystems (although shouldn't that include my 16GB of RAM?)
e) when I try to fill up /tmp I get denied at 12G due to (b) instead of at 19G
f) I should increase capped-memory.swap or mount /tmp with option size=NNNNm where NNNN is something like 8192, to limit it to 8192m which will show in df.

Am I right/wrong?

druuna · 08-18-2010, 03:19 PM

Hi,

I do think gunzip does decompress the whole file first and pipes that to tar.

You actually need even more then I mentioned in post #4: If you have a compressed version (3 Gb), an un-tarred version of the file (4.5 Gb) and extract the archive (another 4.5 Gb) you need at least 12 Gb free space (not sure if both tar.gz and tar will be present the whole time).

The command you use makes it so you can do all tasks in an elegant one-liner instead of first unzipping the file, and then extract the archive and possibly zipping the file again (but that is what happens under the hood). Disregard, I'm wrong!

See post #7 and especially post #10.

AlucardZero · 08-18-2010, 03:27 PM

Hi, "gunzip < filename.tar.gz" puts the data on STDOUT, not on disk. | pipes it to tar. "-f -" makes tar read from STDIN. The .tar file never exists on disk; only the .tar.gz and the {unpacked and uncompressed files} do.

druuna · 08-18-2010, 03:37 PM

You are correct.

I really thought the tar file was created on disk, it is not.

I corrected my previous reply.

Sorry I wasted your time!

jlliagre · 08-18-2010, 03:39 PM

Quote:

c) tmp is mounted on RAM like always

tmpfs is virtual memory based (i.e. RAM + swap space), not RAM only.

Quote:

d) 'df' in a zone shows the host swap size for tmpfs filesystems (although shouldn't that include my 16GB of RAM?)

df shows how much free swap is available unless /tmp is capped with size=xxxx.

Quote:

e) when I try to fill up /tmp I get denied at 12G due to (b) instead of at 19G

yes.

Quote:

f) I should increase capped-memory.swap or mount /tmp with option size=NNNNm where NNNN is something like 8192, to limit it to 8192m which will show in df.

Limiting /tmp capacity would just make the things worse. Increasing capped-memory.swap might help. Extracting elsewhere like on a ZFS filesystem might even help better. How many files are there in the archive ? What is their average size ? tmpfs isn't good for storing a very large number of small files.

jlliagre · 08-18-2010, 03:44 PM

Quote:

Originally Posted by AlucardZero

Hi, "gunzip < filename.tar.gz" puts the data on STDOUT, not on disk. | pipes it to tar. "-f -" makes tar read from STDIN. The .tar file never exists on disk; only the .tar.gz and the {unpacked and uncompressed files} do.

The .tar file doesn't exist on disk as a file but it does use disk space anyway. The pipe lives in virtual memory and that virtual memory is using tmpfs space. Not only the pipe but the input and output buffers used by gunzip and the input buffer used by tar and this might be larger that what you expect.

AlucardZero · 08-18-2010, 03:53 PM

Thanks.

Quote:

Limiting /tmp capacity would just make the things worse.

My thinking is that it would prevent confusion. As long as I set capped-memory.swap large enough, limiting /tmp's size to, say, 8GB would mean it would behave as "df" would lead you to believe: 8GB capacity and full at 8GB, instead of 19GB capacity and full at 12GB.

I also advised the user who initially complained about this to consider /var/tmp instead (which is ZFS; the whole zone is except the default tmpfs filesystems).