get rid of obsolete directories

gunpowder · 07-31-2008, 06:43 AM

Maybe it won't hurt knocking your door again...

what I try to find is a way to delete obsolete subdirs
(obsolete like in: there's only one directory, and no other files):

Example:

Code:

firstsubdir/blahblah/john.txt
secondsubdir/nagnag/1/mike.txt
secondsubdir/nagnag/2/joe.txt
thirdsubdir/keepthis/mary.txt
thirdsubdir/jane.txt

must become:

Code:

firstsubdir/john.txt
secondsubdir/1/mike.txt
secondsubdir/2/joe.txt
thirdsubdir/keepthis/mary.txt
thirdsubdir/jane.txt

All I get with google is how to remove emtpy dirs, been looking a bit further to File::Find to browse the dirtree; but maybe you can push me in the right corner.

Thanks in advance !

bigearsbilly · 07-31-2008, 06:59 AM

here's a starting point for you, if you like

Code:

# true if param directory contains non-directory entries
# ================================================
sub obsolete_dir {

    my $dir = shift;
    opendir DIR, $dir or die "cannee open $dir captain!\n";

    grep { ! -d $_ } readdir DIR;
}

# TEST harness
# print "dir $_ is:", obsolete_dir($_)? "obsolete\n":"ok\n" while $_ = shift @ARGV

jcookeman · 07-31-2008, 10:58 AM

Here you go.

Code:

#!/bin/env bash
#
# Copyright 2008 jcookeman :D
# BSD License
#
# Recursively descend into DIR and remove any superfluous
#+directories. A superfluous directory is a directory 
#+that contains no regular files or is an empty leaf node.
#

function collapse_dirs()
{
     # Set the top level directory.
     TOPLEVEL=${TOPLEVEL:=$1}
     local DIR=$1
     # Get a list of directories.
     local DIRS=`find $DIR -mindepth 1 -maxdepth 1 -type d` >/dev/null 2>&1
     for dir in $DIRS
     do 
          collapse_dirs $dir
     done
     # If TOPLEVEL we cannot do anything past this point.
     if [ "$DIR" = "$TOPLEVEL" ] 
     then
          unset TOPLEVEL
          return 0
     fi
     local FILES=`find $DIR -mindepth 1 -maxdepth 1 -type f` >/dev/null 2>&1
     # If this DIR has no FILES or DIRS, it's an empty leaf node. Delete.
     if [ "${#FILES}" -eq "0" ] && [ "${#DIRS}" -eq "0" ]
     then
          echo "$DIR is an empty leaf node: deleting"
          rmdir $DIR >/dev/null 2>&1
     # If DIR is one below TOPDIR, at this point cannot do anything.
     elif [ "${DIR%/*}" = "$TOPDIR" ]
     then
          return
     # Now DIR has no FILES but child DIRS. Promote them and delete.
     elif [ "${#FILES}" -eq "0" ] 
     then 
          echo "$DIR is an empty directory: collapsing"
          for dir in $DIRS
          do   
               mv $dir ${DIR%/*} >/dev/null 2>&1
               rmdir $DIR >/dev/null 2>&1
          done
     fi
}

COLLAPSE=${1?"No dir specified"}
collapse_dirs $COLLAPSE
exit 0

I just ran it through the full glibc source and it did it pretty quickly. This is on a vm btw...

Code:

[jcook@centosvm ~]$ time ./collapse.sh glibc-2.7
glibc-2.7/sysdeps/unix/sysv/linux/powerpc/powerpc64/power5 is an empty directory: collapsing
glibc-2.7/sysdeps/unix/sysv/linux/powerpc/powerpc64/970 is an empty directory: collapsing
glibc-2.7/sysdeps/unix/sysv/linux/powerpc/powerpc64/power5+ is an empty directory: collapsing
glibc-2.7/sysdeps/unix/sysv/linux/powerpc/powerpc64/power4 is an empty directory: collapsing
glibc-2.7/sysdeps/unix/sysv/linux/powerpc/powerpc64/power6 is an empty directory: collapsing
glibc-2.7/sysdeps/unix/sysv/linux/powerpc/powerpc64/power6x is an empty directory: collapsing
glibc-2.7/sysdeps/unix/sysv/linux/powerpc/powerpc32/power5 is an empty directory: collapsing
glibc-2.7/sysdeps/unix/sysv/linux/powerpc/powerpc32/970 is an empty directory: collapsing
glibc-2.7/sysdeps/unix/sysv/linux/powerpc/powerpc32/power5+ is an empty directory: collapsing
glibc-2.7/sysdeps/unix/sysv/linux/powerpc/powerpc32/power4 is an empty directory: collapsing
glibc-2.7/sysdeps/unix/sysv/linux/powerpc/powerpc32/power6 is an empty directory: collapsing
glibc-2.7/sysdeps/unix/sysv/linux/powerpc/powerpc32/power6x is an empty directory: collapsing
glibc-2.7/sysdeps/unix/sysv/linux/alpha/alphaev67 is an empty directory: collapsing
glibc-2.7/sysdeps/unix/sysv/linux/alpha/alphaev6 is an empty directory: collapsing
glibc-2.7/sysdeps/unix/bsd/bsd4.4/freebsd is an empty directory: collapsing
glibc-2.7/sysdeps is an empty directory: collapsing
glibc-2.7/nptl/sysdeps/unix/sysv is an empty directory: collapsing
glibc-2.7/nptl/sysdeps/unix is an empty directory: collapsing
glibc-2.7/nptl/sysdeps is an empty directory: collapsing

real    0m11.191s
user    0m2.725s
sys     0m7.414s

gunpowder · 08-01-2008, 04:38 AM

Thanks already, I wasn't anywhere near yet

I do see two problems right now
- does not do well with directories containing spaces
- in my example there was a 'firstsubdir/blahblah/john.txt'
This is transformed into 'blahblah/john.txt'
and not 'firstsubdir/john.txt'
(I want to avoid that the name 'blahblah' was already present in the level above)

jcookeman · 08-01-2008, 04:52 AM

Yes there are a couple limitations. The spaces in filenames you can fix by changing

Code:

mv $dir ${DIR%/*} >/dev/null 2>&1

to

mv "$dir" "${DIR%/*}" >/dev/null 2>&1

The other issue is the fact that it will not move directories if the subdir already exists. Actually, that is the case with the example I showed. Many of the sudirectories are simply mirrors of others, so many of them fail.

There are a couple tweaks you can do to the script to make it a bit better. I tossed that together quickly and didn't bother to check everything. One obvious change is that

Code:

local FILES=`find $DIR -mindepth 1 -maxdepth 1 -type f` >/dev/null 2>&1

should be:

# DIRS has probably changed by now. Refresh.
local DIRS=`find $DIR -mindepth 1 -maxdepth 1 -type d` >/dev/null 2>&1
local FILES=`find $DIR -mindepth 1 -maxdepth 1 -type f` >/dev/null 2>&1

Quote:

- in my example there was a 'firstsubdir/blahblah/john.txt'
This is transformed into 'blahblah/john.txt'
and not 'firstsubdir/john.txt'

Of course I didn't implement it that way, because 'firstsubdir' doesn't contain any regular files. If you want to implement it that way, then it shouldn't be that much more difficult. But you have a pretty good basis to tweak.

gunpowder · 08-01-2008, 06:28 AM

OK, I really appreciate your work on this;
I'll try to tweak it a bit, should be much easier than to start from scratch

gunpowder · 08-04-2008, 08:24 AM

This should do it for me:

Code:

#!/bin/env bash
#
# Copyright 2008 jcookeman :D
# BSD License
#
# Recursively descend into DIR and remove any superfluous
#+directories. A superfluous directory is a directory 
#+that contains no regular files or is an empty leaf node.
#

function collapse_dirs()
{
     # Set the top level directory.
     TOPLEVEL=${TOPLEVEL:=$1}
     local DIR=$1
#	 echo "current dir = $DIR"
     # Get a list of directories.
     local DIRS=`find $DIR -mindepth 1 -maxdepth 1 -type d` >/dev/null 2>&1
	 local DIRSno=`cd $DIR >/dev/null 2>&1; ls -l | grep ^d | wc -l` >/dev/null 2>&1
	 
#	 echo "no of subdirs= $DIRSno"

     # If TOPLEVEL we cannot do anything past this point.
     if [ ! "$DIR" = "$TOPLEVEL" ] 
     then

     local FILES=`find $DIR -mindepth 1 -maxdepth 1 -type f` >/dev/null 2>&1
	 local FILESno=`cd $DIR >/dev/null 2>&1; ls -l | grep ^- | wc -l` >/dev/null 2>&1

#	 echo "no of files= $FILESno"
	 
     # If this DIR has no FILES or DIRS, it's an empty leaf node. Delete.
     if [ "$FILESno" -eq "0" ] && [ "$DIRSno" -eq "0" ]
     then
          echo "$DIR is an empty leaf node: deleting"
          rmdir $DIR >/dev/null 2>&1
     # If DIR is one below TOPDIR, at this point cannot do anything.
     elif [ "${DIR%/*}" = "$TOPDIR" ]
     then
          return
     # Now DIR has no FILES but child DIRS. Promote them and delete.
     elif [ "$FILESno" -eq "0" ] && [ "$DIRSno" -eq "1" ] 
     then 
	  echo "$DIR only has one subdirectory: collapse"
          for dir in $DIRS
          do   
			allcontent="/*"
			tomove="$dir$allcontent"
			echo "move = $tomove -> $DIR"
			mv $tomove $DIR >/dev/null 2>&1 
			rmdir "$dir" >/dev/null 2>&1 
		  done
	 fi
#	   unset TOPLEVEL
#          return 0
     fi
	 
	 for dir in $DIRS
     do 
          collapse_dirs $dir
     done

}

COLLAPSE=${1?"No dir specified"}
collapse_dirs $COLLAPSE
exit 0

Still two annoying don't find messages, but not harmful;
can't handle spaces to well in the path/filename;
and need to run this script a few times (something wrong with the recursive loop)

To recreate this example:

Code:

cd;
mkdir firstsubdir;mkdir secondsubdir;mkdir thirdsubdir;
cd firstsubdir;mkdir blahblah;cd blahblah;touch john.txt;
cd ..;cd ..;
cd secondsubdir;mkdir nagnag;cd nagnag;mkdir 1;mkdir 2;cd 1;touch mike.txt;cd ..;cd 2;touch joe.txt;
cd ..;cd ..;cd ..;
cd thirdsubdir;touch jane.txt;mkdir keepthis;cd keepthis;touch mary.txt;
cd