Deleting a Directory with millions of files and sub directories

Habitual · 08-25-2015, 12:25 PM

Isn't bleachbit "fun"?

goumba · 08-25-2015, 12:49 PM

[QUOTE=ramsforums;5410897]

Quote:

Originally Posted by goumba

In addition, sudo doesn't work too well with redirecting and pipes. You need to run that command by using su or sudo bash.

Code:

su -c "nohup /usr/bin/rm -rf <directorypath> >/tmp/nohup.out 2>&1 </dev/null"

Thanks.

If I run on entire directory it may take several hours.
I tried as follows

.. snip... Then executed command as per your suggestion. But the directory is not getting removed.

I copied and pasted a previous poster's response without checking. The command as posted silently fails. D'oh! Try it again, removing /usr (rm is in /bin on my system). Or, even still,

Code:

su -c "nohup $(which rm) -rf <directorypath> >/tmp/nohup.out 2>&1 </dev/null"

$(which rm) will substitute the correct location on your system

rknichols · 08-25-2015, 01:07 PM

Quote:

Originally Posted by jpollard

There are a few. XFS can have "billions"... (it dynamically adds inodes as needed).

In this case, I think it is referring to "more than I can count".

There is a limit to how many links to a file can exist (I don't remember if it is a 16 bit field or a 32 bit field).

For ext4 there is a 65000 link limit, but I don't know whether that's due to the filesystem or a kernel limit for all filesystems.

The top level directory size is about 1 gigabyte, so there can't be "billions" of entries there, but there's been no indication of how deep the tree goes.

ramsforums · 08-25-2015, 01:21 PM

I am not exactly sure billions or millions of directory.

I got the following message for selectively removing folders.

Quote:

rama@develop ~/T6o1L9lGg- $
rama@develop ~/T6o1L9lGg- $ su -c "rm -vfR a*"
Password:
bash: /bin/rm: Argument list too long

ramsforums · 08-25-2015, 01:25 PM

[QUOTE=goumba;5410978]

Quote:

Originally Posted by ramsforums

I copied and pasted a previous poster's response without checking. The command as posted silently fails. D'oh! Try it again, removing /usr (rm is in /bin on my system). Or, even still,

Code:

su -c "nohup $(which rm) -rf <directorypath> >/tmp/nohup.out 2>&1 </dev/null"

$(which rm) will substitute the correct location on your system

Thanks. I have tried as per your suggestion. This too silently failed

Quote:

rama@develop ~ $ su -c "nohup $(which rm) -rvf ~/T6o1L9lGg-/ >/tmp/nohup.out 2>&1 </dev/null"
Password:
rama@develop ~ $ ls
apps Desktop md.log mydelexe.c T6o1L9lGg- Videos
backup Documents Music ocd tempdir xCopy
Copy Downloads mydel.c Pictures Templates
delempty empty mydelexe Public test1
rama@develop ~ $

goumba · 08-25-2015, 01:37 PM

Quote:

Originally Posted by ramsforums

Thanks. I have tried as per your suggestion. This too silently failed

Odd, it worked for me, tested it on several directories.

Code:

which -a rm

If that gives you nothing, there's the problem. Try /bin/rm in it's place.

ramsforums · 08-25-2015, 01:44 PM

Quote:

Originally Posted by goumba

Odd, it worked for me, tested it on several directories.

Code:

which -a rm

If that gives you nothing, there's the problem. Try /bin/rm in it's place.

Thanks. I got as follows

Quote:

rama@develop ~ $ which -a rm
/bin/rm

Now I am executing the following command with su -c option. will update this thread when command completes. in progress now..

Quote:

rama@develop ~ $ su -c "rsync -av --delete ./emptydir/ ./T6o1L9lGg-/"
Password:
sending incremental file list

rknichols · 08-25-2015, 11:02 PM

FWIW, I loaded a directory with ~5,600,000 entries (@ 10,000 links per inode). It took much of the day to fill it, and "rm -r" took 50 minutes to clean it out. This was on an ext4 filesystem on a not terribly fast 320GB disk and with a 1.6GHz AMD CPU.

Lsatenstein · 08-26-2015, 05:02 PM

What about xfs and btrfs and zfs. Are the directory entries in these file systems indexed?

jpollard · 08-26-2015, 05:24 PM

Quote:

Originally Posted by Lsatenstein

What about xfs and btrfs and zfs. Are the directory entries in these file systems indexed?

btrfs uses two methods: btree (good for directory manipulations), and hash (for fast lookups).

I believe xfs uses b+tree.

I don't know about zfs - but I did find reports of poor directory performance with 4000+ files.

Lsatenstein · 08-26-2015, 05:45 PM

Hi JP

Are there any filesystems that use a b+ tree. For a while I thought that btrfs was based on the b+, but I guess I am wrong.

In my old IBM days we used VSAM. (Very successful access method). We joked that it was a ("very special access method).

VSAM was essentially a b+ tree organization, used with spinning disks that provided rotational positional sensing. One dedicated track told the disk controller the realtime angle of the disk. The theory and practice at that time being that the driver would choose the buffer for the sector that was just ahead of the disk head. The software did not have to wait for the disk index marker to be detected before beginning to search to write a buffer.

VSAM brought back memories of control areas, control invervals, free space within each, and also small sized data blocks stored along with the index.

jpollard · 08-26-2015, 06:34 PM

All of that location handling is done in the disk now. The drive itself has the buffer for the given address.