"/" filesystem 100% filled

mackowiakp · 09-28-2019, 12:05 PM

I have a problem with my QNAP (Intel 64bit, 8GB RAM). System software is some bizarre mutation of Ubuntu. For example, filesystem "/" has a capacity of only 290 MB and its normal occupation is 90%. I received such information from QNAP help-desk. The rest of the filesystems (like /var/log, /root, /lib etc. etc) are mounted to "/". Except - surprisingly - /etc. I wrote some of my own scripts and something fills my filesystem "/" to 100%. Does any of colleagues have an idea for some script in the shell (QNAP has no bash, only sh), which, say, every 5 minutes from cron, examines which file on the filesystem has increased its size by, say, 100 Kb (compared to the previous test)? If I detected which file is growing, I would move it elsewhere, e.g. via a symbolic link. Do you have any idea from your colleagues?
Please consider that the total capacity of this filesystem is - as I wrote - only 290 MB. So not much. Hence, such testing of a single file system will not burden the NAS too much.

teckk · 09-28-2019, 12:36 PM

You could use find with -mmin

Find files modified in the last 5 minutes.

Code:

find . -mmin -5 -type f -exec ls -l {} +

find . -mmin -5 -type f | xargs ls -l

You could monitor a file every minute with

Code:

while true; do
    wc -c <filename> | cut -d " " -f1
    sleep 60
done

Save the output as a variable and compare that to next time.

Code:

a=5
b=7
echo $(($b - $a))

Or for sh

Code:

let c="$b-$a"
echo "$c"

Should be able to do that with a few simple small tools.

mackowiakp · 09-28-2019, 12:46 PM

But look at statement:

Code:

find . -mmin -5 -type f -exec ls -l {} +

Dot mean "find recursively from current dir". In my case "the dot" is "/". That means "find" will search thru whole 5 TB disk instead "/" filesystem, 290 MB size only.

pan64 · 09-28-2019, 12:55 PM

find has an option to stay on a single filesystem: -mount
I would probably try:
cd <any dir>; du -sh * | sort -h
but you need to take care about * (or you need to specify the list of dirs).
I think du will work [much] faster.

scasey · 09-28-2019, 01:21 PM

Suggestions are helpful. I'd also try

Code:

du -hd1 /

which will only report the sizes of each directory below /
from man du

Quote:

-d, --max-depth=N
print the total for a directory (or file, with --all) only if it is N or fewer levels below the command line argument; --max-depth=0 is the same as --summarize

Yes, that will report sizes of things not mounted under /, like /home, but it's only going to be one line. You appear to know what's under /, yes?

That said, the first place I'd look would be in /var/log. -- especially if logrotate is not set up/running.

MadeInGermany · 09-29-2019, 05:02 AM

find big recent files on the / filesystem:

Code:

find / -xdev  -size +1000 -mtime -1

List deleted files that are still hold open by a running process;

Code:

lsof | grep '(deleted)'

mackowiakp · 09-29-2019, 08:39 AM

Finally I discovered the cause of the "/" file system overflow. When I create a QNAP offline backup (4 times a day) on another NAS (made from Raspberry Pi, but with a GB port), I sync dnsmasq files at the same time, the source of which is Raspberry Pi. Everything is done via SSH and RSYNC via SSH. Then restart dnsmasq on QNAP, also using the SSH command. This operation causes the /var/log/network/dnsmasqd.log file to grow very fast because QNAP reports some bizarre errors. I used a workaround to reset the contents of this file from cron because they are not needed for anything. Currently, file system occupation is stable at 94%. And this situation QNAP Helpdesk considers correct.

Nevertheless, I think that occupation of file system where logs are stored at level of 95% is wrong. Its crazy, 95% occupation on 290 MB filesystem size, dedicated to logs. Any instability of any application, native or not, leads to a system crash. This should never happen and it is a mistake from my point of view. I reported it to the QNAP Helpdesk and let them think now.

MadeInGermany · 09-30-2019, 01:43 AM

That's why you better put log and spool on a separate file system (in zfs or btrfs simply add directory quota).
Generally I recommend an extra /var

mackowiakp · 09-30-2019, 08:21 AM

spool is less problem. But log - yes. Waiting for native solution from QNAP, I created symbolic link of /var/log to dir on filesystem with huge free space. As a workaround.

smallpond · 09-30-2019, 08:39 AM

Your solution is a good one. Of course, if there is a problem with the disk or filesystem where you have put /var/log, then you will lose the logs. If that is a concern, then you might want to set up some remote logging to another system as well.

mackowiakp · 10-01-2019, 12:57 AM

There is a lot of another strange configs in QNAP. For example size of shared memory is 3,8 GB but usage is less than 150 kB. That means that /dev/shm occupies half of installed memory so it is useless for any apps. I know, default RAM config takes in practically all Linux distros half of memory for tmpfs`s. Of course it could be easy changed in fstab. But QNAP Help-desk informed me that "We do not support units with changed filesys layout". Crazy. They know from me that half of memory is useless. So I adivce him to ask on LinuxQuestions forum if it is too difficult for him.
Whats regarding eventuality of disk failure - there is no problem. My NAS has 4 bays. Each bay is equipped with 5 TB HDD. Each pair of disks is configured as 2 times RAID-1. Disks are hot swapable. So if HDD failure accrues, it is easy to repair such situation without data lost.

rknichols · 10-01-2019, 09:06 AM

Quote:

Originally Posted by mackowiakp

There is a lot of another strange configs in QNAP. For example size of shared memory is 3,8 GB but usage is less than 150 kB. That means that /dev/shm occupies half of installed memory so it is useless for any apps.

No, it doesn't. That 3,8GB is a maximum. At any given time, /dev/shm uses only the memory it needs. The same is true for other tmpfs filesystems. Right now I have tmpfs filesystems with allocations that total more than the size of my memory. The system runs fine. Actual tmpfs usage is less than 1% of memory.