File exists and yet it doesn't

dreaken667 · 07-11-2011, 03:57 PM

Hello all.

I have a rather strange issue I'm attempting to fix here. One of the servers I work with had a log file reach 2GB in size and stopped logging. The software which is logging to said file cannot be stopped or restarted and does not provide a way to rotate log files -- poor design, I know. So in my infinite wisdom (or stupidity, however you want to read that), I truncated the log file in place with:

Code:

>vmsproxy.log

When I realized the log file was not being appended to, I checked lsof output and it shows the following:

Code:

vmsproxy  11541   root    2w   REG         9,6          0     295029 /var/log/vmsproxy.log (deleted)
vmsproxy  12779   root    2w   REG         9,6          0     295029 /var/log/vmsproxy.log (deleted)
vmsproxy  12782   root    2w   REG         9,6          0     295029 /var/log/vmsproxy.log (deleted)
vmsproxy  12783   root    2w   REG         9,6          0     295029 /var/log/vmsproxy.log (deleted)
vmsproxy  12784   root    2w   REG         9,6          0     295029 /var/log/vmsproxy.log (deleted)
vmsproxy  12785   root    2w   REG         9,6          0     295029 /var/log/vmsproxy.log (deleted)
vmsproxy  12786   root    2w   REG         9,6          0     295029 /var/log/vmsproxy.log (deleted)
vmsproxy  12787   root    2w   REG         9,6          0     295029 /var/log/vmsproxy.log (deleted)
vmsproxy  12788   root    2w   REG         9,6          0     295029 /var/log/vmsproxy.log (deleted)
vmsproxy  12790   root    2w   REG         9,6          0     295029 /var/log/vmsproxy.log (deleted)
vmsproxy  12791   root    2w   REG         9,6          0     295029 /var/log/vmsproxy.log (deleted)
vmsproxy  12793   root    2w   REG         9,6          0     295029 /var/log/vmsproxy.log (deleted)

So I tried to recover the file by running:

Code:

cp /proc/11541/fd/2 /var/log/vmsproxy.log

The problem I have now is this:

Code:

# cat vmsproxy.log
cat: vmsproxy.log: No such file or directory
# file vmsproxy.log
vmsproxy.log: can't stat `vmsproxy.log' (No such file or directory).
# ls -i vmsproxy.log
ls: vmsproxy.log: No such file or directory
# touch vmsproxy.log
touch: creating `vmsproxy.log': File exists
# rm -f vmsproxy.log
# touch vmsproxy.log
touch: creating `vmsproxy.log': File exists
# lsattr vmsproxy.log
lsattr: No such file or directory while trying to stat vmsproxy.log

The file exists and yet it doesn't. Meanwhile, I have no logfile to perform troubleshooting with. My question is this:

Does anybody know how I can make this file either go away or come back? I'll be more than happy to provide command output.

jefro · 07-11-2011, 04:16 PM

Run a filesytem check on the drive(s)

T3RM1NVT0R · 07-11-2011, 04:17 PM

Hi there,

Type the output of ls -al /var/log

What does

Quote:

rm -r -f /var/log/vmsproxy.log

yeilds?

dreaken667 · 07-11-2011, 04:19 PM

I was afraid you were going to say that. I cannot take the system down to run fsck for the same reason I could not restart the service creating the log file. It's a production system that is under heavy load 24/7.

EDIT:

rm -rf /var/log/vmsproxy.log yeilds no output
ls -al /var/log prints the contents of the directory, but vmsproxy.log is not in the list

SL00b · 07-11-2011, 04:23 PM

If this is a heavy-use, business critical, 24/7 system, why do you not have a second system you can failover to?

dreaken667 · 07-11-2011, 04:28 PM

I didn't design the system or the network. I've taken the position of someone that used to work here.
A backup system has been proposed to the higher ups, but there is not one at the moment.

SL00b · 07-11-2011, 04:47 PM

I guess you just got some ammunition for that fight, then.

It looks to me like user vmsproxy had the file open when it was deleted, and it still has it open even though it's gone. The easiest way to resolve that would be to shut down the application, and you might even see those (deleted) issues disappear right away. If not, you should at least be able to remove them with the app down, restart, and all is kosher. If this works out, we're talking about a 5 minute outage.

And if it doesn't... then however long it takes to perform an fsck and act on the results.

dreaken667 · 07-11-2011, 06:00 PM

I had the ammunition to begin with, but this is definitely icing on the cake.
That confirms my suspicions then. Thank you all for the time.