Delete millions of files from a directory
 

I just encountered a nasty bug in the Acronis Cloud software. The bug seems to produce an indefinite number of log files in /var/lib/Acronis/msp/zmq/logs. They've since came out with a patch for this, but I'm still stuck with a directory full of millions of logs that I cannot delete. I'm estimating there are about 15 million files in that directory, but I'm not even sure about that. Honestly for all I know, there could be 150 million.

I can't even touch this directory. Any command I try (ls, rm, find) hangs where this directory is concerned. I've tried all of the suggestions here with no luck: http://www.slashroot.in/which-is-the...files-in-linux

I let each of the methods mentioned in that article run for at least several hours. When I would come back and run `df -h` in a separate shell, there was no change in the disk usage. On the rsync method, I tried adding `-v` to see if it was actually doing anything, but the only line I got was "sending incremental file list". This was last night, and the operation continued to hang like that when I came back and checked on it this morning.

If it matters, this machine is running CentOS 5 and the file system type is ext3. Everything except /boot is installed on one root (/) LVM volume.

Is there any way to resolve this or am I better off backing up the important data and reinstalling?

First thing I'd do, would be to stop/kill the Acronis process. If the files are open when they're 'deleted', the disk space will still remain in use. Stopping the process may help. It may also let you do the "rm -fR" on that directory.

I've had that too-many-files problem before, and sadly, have had to hack at it sometimes, deleting 'chunks' of files that match a smaller pattern, to get things down a bit. Something like "rm *2345.log", etc. (you see where I'm going), then do *2346.log...lather-rinse-repeat. Sometimes that's the only way.

Yep! You and I think alike!

That was one of the first things I tried, with no luck.

I think this will have to be my next attempt. Unfortunately, since I can't read the directory, all I have to go on are the example file names the end user gave me when they reported the issue.

Thanks for your response, I'll keep this thread updated on what happens.

So you did stop it, and disk space stayed up?
I figured you did, since that was on the list from that link, but I didn't know if you had tried it after stopping the Acronis service. And out of curiosity...did you try renaming the directory???
Yeah, I've hated having to do that in the past, but sometimes you're left with no choice.

Yeah, there were three Acronis services running. We stopped all three of them and disk space didn't change.

I didn't try renaming the directory. Why would I want to try that?

Yeah, I'm still trying to confirm the pattern of file name I would use. The example the user gave me when they were able to read the directory is "client_session-libzmq_infra-2016-01-01-20-01-28-199.log". I tried deleting that one file in particular using rm, but rm just hung for probably about 30 seconds before telling me there was no such file or directory. This probably means the user already deleted that particular file. Now I'm trying `find . -name client_session-libzmq_infra-2016*` to try and confirm the pattern, but it's just hanging.

If you don't have that "*" quoted, your shell is trying to expand that wildcard, which requires reading through the entire directory, finding all the names that match the pattern, and then sorting the resulting list. If you put that pattern in quotes, then it is the find process that reads through the directory and processes each matching name as it finds it.

Thanks for the tip. I've killed my existing operation and used your example. Still not showing any results, but here's hoping.

Can you explain what this does? I'm not familiar with Perl at all...

It deletes all files in /var/lib/Acronis/msp/zmq/logs/

map is a function that apply an expression for each element of a list (it's like a loop)
So for each element in /var/lib/Acronis/msp/zmq/logs/*, it unlinks it (deletes it) or warn if there is an error trying to delete that file

+1 for the perl one-liner. Always love them.

I've reached the point where I'm giving up. I've spent hours attempting to delete these files, when the server is due for an upgrade/migration anyway, so we're just going to move the data to a newer server and decommission it.

I appreciate all the suggestions. I didn't get a chance to try that Perl command, but I'll definitely keep it in mind if something like this happens again.

I'm not sure whether or not to mark this thread as SOLVED, since I didn't really solve the problem. I'll leave that up to a moderator.

Heck, I'd give the perl one-liner a shot just to see what it does. At this point, you've got nothing to lose. :)

I would have liked to, but this isn't a machine owned or managed by me. The decision was that of the user who owned it and I no longer have access to it.