I have been working on this for a few hours now and still haven't been able to track down whats going on.
I just switched servers and put a fresh debian install on with apache2 + custom suexec + php5(fcgi).
now when i upload a file to the site it wont upload, so i tracked down the issue. its the /tmp directory that root tmp. its at 100% so i restart the /etc/init.d/apache2 process and the /tmp directory is automagically cleared to 20% used and i can upload again. so i figure ok its got to be a setting. i looked in all the conf files conf.d the php5 conf files and no where do i have a /tmp directory set. so i think OK I can build a cron to clear the tmp directory every 10 mins till i find the issue, but alas, when i go into the tmp directory There are NO files to delete!!! wtf. holy crap. and all that goodness, this is crazy, how can you have a tmp directory that is not being written to, from all i can see, so i think ok it might be a hard link, i checked, in all root and sub root directories no hard links to tmp, no soft links to temp either. and if i sit in my /tmp directory and do a df -h after some uploads its going from 20% so 23% and up, but i do ls -alh and nothing, then ls -al and watch bytes, nothing , so this is very strange. has anyone encountered something like this? is it possible that this is a hard link thats hidden somewhere or some other extreme function i have never heard of?
thanks for anyone's help, i really appreciate it,
-=Levi=-

Let's see the output of: $ df -h

Restarting httpd every 10 minutes is not a good solution, BTW.

Yeah I know, but i dont have the time to sit there and df -h every few minutes to know when to reset apache, there are users uploading all the time unfortunately, so tmp gets loaded up quick. i am using a apache2 mod-worker with fcgi/php5 setup so i was thinking it could be something in php5 that is using tmp, but the lack of any files appearing in /tmp is very very strange.

Filesystem Size Used Avail Use% Mounted on
/dev/sda4 64G 2.2G 59G 4% /
tmpfs 4.0G 0 4.0G 0% /lib/init/rw
udev 10M 88K 10M 1% /dev
tmpfs 4.0G 0 4.0G 0% /dev/shm
/dev/sda1 190M 16M 166M 9% /boot
/dev/sda2 479M 185M 270M 41% /tmp
/dev/sdb1 459G 215G 221G 50% /home

Oh and so you know there are no files in /tmp to be deleted, the only thing that seems to clear out /tmp is to run /etc/init.d/apache2 restart. as well if i add files to /tmp, the df -h shows the increase in usage.... soo freakin weird. I != grock(apache2 tmp)

du -h /tmp might reveal something different from df

Check for open files in the tmp directory. Something like lsof |grep tmp. It can be that the apache still has the files in use.

du -h didnt return anything of note, it had a few 10k files but nothing in the megs, but % on temp was 40% of 480M so i did the lsof | grep tmp and got quite an interesting return

php5-cgi 26793 www-data DEL REG 8,2 2110 /tmp/xcache
php5-cgi 26793 www-data 3u REG 8,2 0 2111 /tmp/.xcache.33.0.1415126335.lock (deleted)
php5-cgi 26793 www-data 4u REG 8,2 0 2112 /tmp/.xcache.33.1.1197600172.lock (deleted)
php5-cgi 26793 www-data 5u REG 8,2 0 2113 /tmp/.xcache.33.2.1624911851.lock (deleted)
php5-cgi 26793 www-data 6u REG 8,2 0 2114 /tmp/.xcache.33.3.829331906.lock (deleted)
php5-cgi 26793 www-data 7u REG 8,2 0 2115 /tmp/.xcache.33.4.508711270.lock (deleted)
php5-cgi 26793 www-data 8u REG 8,2 0 2116 /tmp/.xcache.33.5.783983198.lock (deleted)
php5-cgi 26793 www-data 9u REG 8,2 0 2117 /tmp/.xcache.33.6.631367521.lock (deleted)
php5-cgi 26793 www-data 10u REG 8,2 0 2118 /tmp/.xcache.33.7.921194865.lock (deleted)
php5-cgi 26793 www-data 11u REG 8,2 0 2119 /tmp/.xcache.33.8.153613356.lock (deleted)
php5-cgi 26793 www-data 12u REG 8,2 0 2120 /tmp/.xcache.33.9.1394068807.lock (deleted)
php5-cgi 26793 www-data 13u REG 8,2 0 2121 /tmp/.xcache.33.10.1995613073.lock (deleted)
php5-cgi 26793 www-data 14u REG 8,2 0 2122 /tmp/.xcache.33.11.41708870.lock (deleted)
php5-cgi 26793 www-data 15u REG 8,2 0 2123 /tmp/.xcache.33.12.1753950792.lock (deleted)
php5-cgi 26793 www-data 16u REG 8,2 0 2124 /tmp/.xcache.33.13.1805624952.lock (deleted)
php5-cgi 26793 www-data 17u REG 8,2 0 2126 /tmp/.xcache.33.14.1695875822.lock (deleted)
php5-cgi 26793 www-data 18u REG 8,2 0 2128 /tmp/.xcache.33.15.1883520580.lock (deleted)


it appears xcache isnt cleaning up after itself however i cannot see these hidden files in tmp. and the list says (deleted) so how does that work? im gonna check my settings in xcache and see if i can tweak it a bit to stop giving this error.

it was xcache in php5, so i went into the xcache settings and changed the xcache.mmap_path to the tmp dir in home.
thanks for that lsof hint there, shoulda though of that much thanks.

Glad you got it resolved. As a side note, your /tmp is obviously too small for a server. If you're using LVM (and have extra space), you might consider increasing it to ~1GB.

yar, I'm gonna do that, the hosting company we go with actually setup the partitions, so i have to reconfigure on new setups, i actually get to have some fun tomorrow, getting a load balanced setup with a bunch more servers. definitely going to rock. luckily i figured this one out before moving it to a large setup that coulda been waayyyy too much fun.