[SOLVED] files created then deleted at every second in tmp directory

adrhc · 04-02-2019, 11:49 AM

By mistake I noticed that in /tmp directory are continuously created some files then immediately deleted. Using a succession of ls -l /tmp I managed to catch the created files:

Code:

-rw------- 1 root root       0 Apr  2 19:37  YlOmPA069G
-rw------- 1 root root       0 Apr  2 19:37  l74jZzbcs6

or another example:

Code:

-rw------- 1 root root       0 Apr  2 19:44  AwVhWakvQ_
-rw------- 1 root root       0 Apr  2 19:44  RpRGl__cIM
-rw------- 1 root root       0 Apr  2 19:44  S0e72nkpBl
-rw------- 1 root root       0 Apr  2 19:44  emxIQQMSy2

It's about Ubuntu 18.10 with 4.18.0-16-generic. This is an almost fresh install: I added some server software (nginx, mysql, php7.2-fpm) but even with those closed the problem persists.

What are the files created and why?
How would I stop this behaviour? a very undesirable one on a SSD

Thank you!

BW-userx · 04-02-2019, 11:54 AM

you need to find the app doing this. what you got running outside of standard system, if not, then go look at standard system apps. try putting anotime in your fstab for your mounts if it is sdd. man anotime.

MensaWater · 04-02-2019, 12:13 PM

You could try running "lsof /tmp" to see if you can capture file names like you did with "ls -l". The lsof otuput will show show you the PID in the second column. You can run "ps -ef |grep <PID>" to see more info about the process that has the specified PID.

ondoho · 04-02-2019, 01:10 PM

Quote:

Originally Posted by adrhc

How would I stop this behaviour? a very undesirable one on a SSD

you can make your /tmp a tmpfs, if it isn't one already.

rknichols · 04-02-2019, 01:42 PM

Quote:

Originally Posted by adrhc

By mistake I noticed that in /tmp directory are continuously created some files then immediately deleted.
...
How would I stop this behaviour? a very undesirable one on a SSD

Really, small files that are created and soon deleted are usually never written to the drive. They exist in the kernel buffers, and are then deleted (freeing the buffer) before the dirty buffer is ever flushed out to the drive. For that sort of usage, /tmp in a tmpfs in memory really isn't much different from /tmp on a physical drive. Files in a tmpfs will get pushed out to swap if there is memory pressure, and dirty buffers will be flushed out sooner for the same reason. There is a difference for long-lived files, since unreferenced pages in a tmpfs can sit around for a very long time without being flushed out, whereas the dirty buffer flush is more aggressive.

syg00 · 04-03-2019, 01:12 AM

If they show up in a "ls", are they hitting the SSD, which the OP seems to want to avoid. Personally I always put /tmp on tmpfs.
As for the files themselves, I'd be real interested in finding out what is allocating them. In Fedora I see names like dnf-asdvav or systemd-tmpfiles.gtejgbnb, so the files give an indication themselves.
@adrhc go get fatrace and run it using sudo - it will tell you what process is hitting each file. Very handy for a quick check.

MensaWater · 04-03-2019, 07:56 AM

Quote:

Originally Posted by ondoho

you can make your /tmp a tmpfs, if it isn't one already.

Presumably the OP is concerned because of SSD "wear" but that really isn't as big a concern as it once was.
https://www.cnet.com/how-to/how-ssds...ease-lifespan/

Since SSD uses flash memory the benefit to using tmpfs in memory isn't that great from a performance standpoint. It also has the downside of using up system memory for storage that might be used for processes instead. Also of course the same "wear" that would affect SSD drives would affect the DIMMs you're using for tmpfs. Using tmpfs makes sense if you're looking for performance increases on systems that have mechanical drives but not as much if you're using SSD or Flash and none at all if you're using NVMe.

We used SSD disks in a Hitachi disk array for several years and never saw performance degradation. A couple of years back we switched to a Pure Flasharray using flash disks and also haven't seen any degradation.

syg00 · 04-03-2019, 08:11 AM

Hmmm - tmpfs uses kernel cache, and doesn't use a filesystem as generally accepted. It is always ahead of anything else performance wise. Whether any of us can measure such benefit in everyday usage is an entirely different discussion.

rknichols · 04-03-2019, 08:38 AM

Quote:

Originally Posted by syg00

If they show up in a "ls", are they hitting the SSD,

That is not correct. The ls command is simply querying the kernel's view of the filesystem, and does not directly access the drive.

rknichols · 04-03-2019, 08:47 AM

Quote:

Originally Posted by MensaWater

Since SSD uses flash memory the benefit to using tmpfs in memory isn't that great from a performance standpoint. It also has the downside of using up system memory for storage that might be used for processes instead. Also of course the same "wear" that would affect SSD drives would affect the DIMMs you're using for tmpfs.

Everything that is read from or written to the SSD is going through the kernel's buffer cache, so the "wear" on the DIMMs (if any) would be the same. For a tmpfs, that's where the data stays unless pushed out to swap because processes need the memory. For a regular filesystem, the data stays in the buffer cache until the kernel gets around to flushing out the dirty buffer, and that can come sooner if there is memory pressure. There really isn't much difference for short-lived files.

pan64 · 04-03-2019, 02:05 PM

if I remember well the tmpfs is much faster than an ssd. Speaking about performance: compiling/building something huge will show you the difference (if compiler uses /tmp). But obviously you need enough RAM to do that.

adrhc · 04-04-2019, 09:45 AM

Hi, I just copy the answer from https://askubuntu.com/questions/1130...-tmp-directory:

Quote:

I suggest installing and running fnotifystat to detect the process that is creating these files:

sudo apt-get install fnotifystat
sudo fnotifystat -i /tmp
You will see process that is doing the open/close/read/write activity something like the following:

Total Open Close Read Write PID Process Pathname
3.0 1.0 1.0 0.0 1.0 5748 firefox /tmp/cubeb-shm-5748-input (deleted)
2.0 0.0 1.0 0.0 1.0 18135 firefox /tmp/cubeb-shm-5748-output (deleted)
1.0 1.0 0.0 0.0 0.0 5748 firefox /tmp/cubeb-shm-5748-output (deleted)

MensaWater · 04-04-2019, 10:43 AM

Quote:

Originally Posted by adrhc

Hi, I just copy the answer from https://askubuntu.com/questions/1130...-tmp-directory:

So did you run that and determine what was creating the files?

Did you ever try doing it with lsof as I'd suggested?

adrhc · 04-05-2019, 01:27 AM

I tried fnotifystat and works really well: it continuously prints the files but after first or second iteration is pretty easy to spot the problem (file & pid). First fnotifystat iteration:

Code:

$sudo fnotifystat -i /tmp
Total   Open  Close   Read  Write   PID  Process         Pathname
  3.0    1.0    1.0    0.0    1.0  17052 <unknown>       /tmp/MjBDibAgj4
  2.0    0.0    1.0    0.0    1.0   1058 perl            /tmp/KvNyYEK0o1 (deleted)
  2.0    0.0    1.0    0.0    1.0  17052 <unknown>       /tmp/NuIwKpE5fW (deleted)
  2.0    0.0    1.0    0.0    1.0   1058 perl            /tmp/bqs_GgDZr3 (deleted)
  2.0    1.0    0.0    1.0    0.0  17052 <unknown>       /tmp/hVNxfwGiA5
  2.0    0.0    1.0    0.0    1.0  17052 <unknown>       /tmp/hVNxfwGiA5 (deleted)
  2.0    0.0    1.0    0.0    1.0  17052 <unknown>       /tmp/sBfHsaEvBa (deleted)
  1.0    1.0    0.0    0.0    0.0   1058 perl            /tmp/KvNyYEK0o1
  1.0    1.0    0.0    0.0    0.0  17052 <unknown>       /tmp/NuIwKpE5fW
  1.0    1.0    0.0    0.0    0.0   1058 perl            /tmp/bqs_GgDZr3
  1.0    0.0    0.0    0.0    1.0  17053 <unknown>       /tmp/hVNxfwGiA5
  1.0    1.0    0.0    0.0    0.0  17052 <unknown>       /tmp/sBfHsaEvBa

I also tried lsof which I guess should work too but it's very difficult to catch those short lived files; as a matter of fact I never managed to catch one (at least visually if not with a select & copy).

watch --interval=1 'lsof /tmp/'

Code:

Every 1.0s: lsof /tmp/                                                                                         adrhc.go.ro: Fri Apr  5 09:32:46 2019

lsof: WARNING: can't stat() hugetlbfs file system /var/lib/hugetlbfs/group/www-data/pagesize-2MB
      Output information may be incomplete.
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java      4013  gigi  mem    REG   0,49    32768   36 /tmp/hsperfdata_adr/4013
clementin 5319  gigi   13uW  REG   0,49        0   43 /tmp/qtsingleapp-clemen-d211-3e8-lockfile

6th ls -l /tmp iteration:

Code:

$ll /tmp/
total 44K
drwxrwxrwt 2 root root      40 Apr  5 08:03 VMwareDnD
drwx------ 2 root root      40 Apr  5 08:03 cvcd
drwxr-x--- 2 gigi  gigi       60 Apr  5 08:03 hsperfdata_adr
drwxr-xr-x 2 gigi  gigi       40 Apr  5 08:03 pms-3956e063-70fc-4db0-86e0-8fdb4233df78
drwx------ 2 root root      40 Apr  5 08:03 pulse-PKdhtXMmr18n
drwx------ 2 gigi  gigi       60 Apr  5 08:06 ssh-cjqCKNGDzviv
drwx------ 2 gigi  gigi       60 Apr  5 09:19 ssh-gfARQxaf6m
drwx------ 3 root root      60 Apr  5 08:03 systemd-private-63b3741072be42e99d6745a3cb7ea09a-ModemManager.service-iqOTmw
drwx------ 3 root root      60 Apr  5 08:06 systemd-private-63b3741072be42e99d6745a3cb7ea09a-rtkit-daemon.service-cVfjYe
drwx------ 3 root root      60 Apr  5 08:03 systemd-private-63b3741072be42e99d6745a3cb7ea09a-systemd-timesyncd.service-EqKUMa
drwx------ 3 root root      60 Apr  5 08:06 systemd-private-63b3741072be42e99d6745a3cb7ea09a-upower.service-aArT4h
drwx------ 2 root root     160 Apr  5 08:03 vmware-root
-rw------- 1 root root       0 Apr  5 09:30 2ArUcC_n0m
-rw------- 1 root root       0 Apr  5 09:30 7LJHogPfCx
-rw------- 1 root root       0 Apr  5 09:30 N31uTa3UBB
-rw------- 1 gigi  gigi        0 Apr  5 08:05 config-err-mj0nEn
-rw-r--r-- 1 gigi  gigi      42K Apr  5 08:05 fake_useragent_0.1.11.json
srwxrwxrwx 1 root www-data   0 Apr  5 08:03 nginx-rtmp.0
srwxrwxrwx 1 root www-data   0 Apr  5 08:03 nginx-rtmp.1
srwxrwxrwx 1 root www-data   0 Apr  5 08:03 nginx-rtmp.2
srwxrwxrwx 1 root www-data   0 Apr  5 08:03 nginx-rtmp.3
srwxrwxr-x 1 gigi  gigi        0 Apr  5 08:06 qtsingleapp-clemen-d211-3e8
-rw-rw-r-- 1 gigi  gigi        0 Apr  5 08:06 qtsingleapp-clemen-d211-3e8-lockfile
-rw------- 1 root root       0 Apr  5 09:30 xYyGd3nrWc

ondoho · 04-05-2019, 12:13 PM

there's only two pids involved: 17052 and 1058.
the processes those belong to should be easy to find with ps or top or htop.