Useful custom commands for Slackware

gbonvehi · 07-07-2004, 05:09 AM

Try changing the line:
cat /var/log/packages/$( echo "$list" | grep -i "$pkg" ) | less ;;

to

cat /var/log/packages$( echo "$list" | grep -i "$pkg" ) | less ;;

thegeekster · 07-07-2004, 01:45 PM

carboncopy

Thanx for pointing out this error. gbonvehi has the right line that needs changing, which is line number 53 (the third from the last line, above the last two "esac" lines).................However, to correct it simply remove the path "/var/log/packages/" that follows the word "cat"...........it should read:

cat $( echo "$list" | grep -i "$pkg" ) | less ;;

OR you can use:

cat `echo "$list" | grep -i "$pkg"` | less ;;

NOTE: In bash scripting, both lines above are identical. The use of backquotes (not to be confused with the apostrophe) has the same meaning as using parenthesis with the dollar symbol in front of it.......I've changed my style to use the backquotes, ` ` , instead of the dollar sign/parentheses construction, $( ), because it's easier for me to type (On US keyboards, the backquote key is the one above the Tab key, to the left of the number 1 key).......

thegeekster · 07-07-2004, 08:21 PM

Okay, let's try this again.............for the "pkginfo" script, there needs to be another change.............I know what happened.........I changed the lookup from 'ls /var/log/packages/*' to "find /var/log./packages/* -prune -type f" which changed the output to include the full path....................

So _all_ the necessary change will be as follows (changes made according to the line number):

Line 37

Code:

Change:
          cat /var/log/packages/$list | less	;;

to this:
          cat $list | less	;;

Line 47

Code:

Change:
`echo -e "$list\n" | more`

to this:
`echo -e "$list\n" | while read i ; do basename $i ; done | more`

Line 53 (this is the change I mentioned above)

Code:

Change:
        cat /var/log/packages/$( echo "$list" | grep -i "$pkg" ) | less ;;
 
to this:
        cat `echo "$list" | grep -i "$pkg"` | less ;;

NOTE: i've already made the appropriate changes in the top post above......Sorry about this, was in too much of a hurry....

carboncopy · 07-07-2004, 10:08 PM

Thanks for the Great Code. save a tonne of time (pkgtool --> view pkg).

Bebo · 07-08-2004, 03:05 AM

OK, I haven't tried your new scripts yet (I will after finishing this post though

), but I just wanted to say that if you can make the orphan script work I'd be very interested to see and use it. I tried to write exactly that kind of script but I must have given it too little thought 'cause it just was far too slow and memory consuming. In my script I also looked for "runaways", i.e. files belonging to an installed package that just weren't on the computer - perhaps something for your script too?

thegeekster · 07-08-2004, 04:22 AM

Bebo

It will probably take a bit of time to check the system files as it is, I don't really see any way around that, so I was only going to target the normal directories most apps install programs to...........And try to avoid any data directories or partitions.............I made a backup script a while back that is able to read /etc/fstab and /etc/mtab to find what system directories are mounted (such as /boot, /home /usr, etc.), mount the /boot directory if it isn't mounted {and unmount afterwards when done}...........And to exclude any mounted directories that are not the system directories (usually being the data directories)............

And yes, I was going to verify that what a program installed is still there........Basically, that would entail doing a check to see if the file exists by running through the lists of files in the /var/log/packages/ directory ( if [ -f FILE ] ; then ... ) and flagging the ones that are missing.............I was also going to use part of the code from my "whichpkg" script to verify the symlinks installed by programs as well...................

And carboncopy........thanx for the vote of confidence.....

Bebo · 07-09-2004, 05:21 AM

I couldn't help myself, so I rewrote my orphans script from scratch. I hope you don't mind if I post it here. It's rather long, so pardon me for clogging this thread :)

Note: This script might still be subject to updates, which will then replace the script here in this very post together with update info in a new post. Info on most recent update is posted below.

BEWARE: the script can of course NOT be used blindly - you HAVE to double-check the results! By this I mean that you have to make sure that the files reported aren't used/needed by some program. Although the script checks in both /var/adm/packages/* as well as /var/adm/scripts/* there will be false-positives, such as files generated by programs. These often contain important configuration, and other, data. For instance, some (most?) of the files in /etc that are reported as orphans might be extremely important.

Anyways, here it is.

Code:

#!/bin/sh

DEFAULTDIRS=( /bin /etc /opt/kde /sbin /usr )
ORPHANSKIPDIRS=( /dev /proc /sys /tmp /usr/tmp /usr/src /var /lib )
RENEGADESKIPDIRS=( /dev /lib /usr/src )


LOGCOMMAND=$(echo "$(basename $0) $@                          " | cut -c1-26 | tr -d '\n')
LOGFILE=$HOME/$(basename $0).log

ORPHANFILES=$PWD/orphaned-files
ORPHANTREE=$PWD/orphaned-dirs
RENEGADELIST=$PWD/renegades

TMPDIR=$(mktemp -d /tmp/$(basename $0)-XXXXXX) || exit 2

SLACKFILES=$TMPDIR/slackfiles
SLACKDIRS=$TMPDIR/slackdirs

DIRTREE=$TMPDIR/dirtree
ORPHANDIRS=$TMPDIR/orphandirs

PACKAGEFILES=$TMPDIR/packagefiles
PACKAGEDIRS=$TMPDIR/packagedirs

PKGFILES=$TMPDIR/pkgfiles
DOTNEWFILES=$TMPDIR/dotnewfiles
TEMPFILE=$TMPDIR/tempfile



renice 20 -p $$ > /dev/null

trap cleanup SIGINT

IFS=$'\n'



#======================================================================
# DEFINE FUNCTIONS
#----------------------------------------------------------------------
usage ()
{
cat << EOF

NAME
   $(basename $0) - script for locating orphaned and identify renegade files.

SYNOPSIS
   $(basename $0) [ACTIONS] [OPTIONS] [DIRECTORY...] 

DESCRIPTION
   $(basename $0) will search for both orphaned files and directories as well 
   as renegade files. An orphan is a file/directory on the file system that do 
   not belong to any package as listed in /var/adm/packages. Renegade files are
   files that belong to installed packages but can not be found at the correct 
   location. 

   Directories subject to a search can be given as arguments on the command 
   line. Some directories should usually be omitted in a search; these are 
   given in the arrays ORPHANSKIPDIRS and RENEGADESKIPDIRS at the top of the 
   script. These are currently:
      ORPHANSKIPDIRS:   ${ORPHANSKIPDIRS[@]}
      RENEGADESKIPDIRS: ${RENEGADESKIPDIRS[@]}

OUTPUT
   The results of an orphan scan are two files containing respectively one list 
   of files and one list with a collapsed directory tree. The list files are 
   defined by the variables ORPHANTREE and ORPHANFILES. These are currently:
      ORPHANTREE:  $ORPHANTREE
      ORPHANFILES: $ORPHANFILES

   A renegade search will put the results in RENEGADELIST, currently set to
      RENEGADELIST: $RENEGADELIST

DEFAULT
   If no action is given on the command line, searches for both orphans and 
   renegades are conducted. 

   If no directories are given as arguments on the command line, the search 
   will be conducted in a number of default directories, given as the array 
   DEFAULTDIRS at the top of the script. These are currently: 
      DEFAULTDIRS: ${DEFAULTDIRS[@]}

ACTIONS
   --orphans | -o
      Search for orphaned files.

   --renegades | -r
      Search for renegade files.

OPTIONS
   --prune | -p 
      Do not descend below the given directory.

   --quiet | -q 
      Disable logging to logfile.

EOF
}



cleanup ()
{
    [ x$1 == x ] && echo -e "\n\n   SIGINT caught. Cleaning up.\n"
    
    case "$1" in
	orp* ) rm "$ORPHANFILES" "$ORPHANTREE" 2> /dev/null ;;
	ren* ) rm "$RENEGADELIST" 2> /dev/null ;;
        *    ) TEMP="$(echo $TMPDIR | sed 's|/tmp||' | sed 's|^/||')" && [ "x$TEMP" != x ] && ( cd /tmp && rm -rf "$TEMP" ) ;;
               # ^^ MAKE SURE WE DON'T DO SOMETHING REALLY STUPID ^^
    esac

    [ x$1 == x ] && exit 1
}



report ()
{
    REPORT="$1"
    ACTION="$2"
    STARTDATE="$3"
    ELAPSED="$4"

    case "$REPORT" in
	screen )
	    case "$ACTION" in
		orp* )
		    printf "   %4d and %4d orphaned directories and files found and listed in %s and %s.\n" \
			"$(cat $ORPHANTREE 2> /dev/null | grep '^/' | grep -c .)" "$(cat $ORPHANFILES 2> /dev/null | grep -c .)" "$ORPHANTREE" "$ORPHANFILES"
		    ;;
		ren* )
		    printf "   %4d renagade files detected and listed in %s.\n" \
			"$(cat $RENEGADELIST 2> /dev/null | grep -c .)" "$RENEGADELIST"
		    ;;
	    esac
	    ;;
	file )
	    case "$2" in
		orp* )
		    LOGLINE=$(printf "%s ORPHAN   search %s; %4d/%4d orphaned directories/files found in %8.3f seconds.\n" \
			"$LOGCOMMAND" "$STARTDATE" "$(cat $ORPHANTREE 2> /dev/null | grep '^/' | grep -c .)" \
			"$(cat $ORPHANFILES 2> /dev/null | grep -c .)" "$ELAPSED")
		    ;;
		ren* )
		    LOGLINE=$(printf "%s RENAGADE search %s;      %4d renagades detected in               %8.3f seconds.\n" \
			"$LOGCOMMAND" "$STARTDATE" "$(cat $RENEGADELIST 2> /dev/null | grep '^/' | grep -c .)" "$ELAPSED")
		    ;;
	    esac
	    [ "x$LOGFILE" != x ] && echo "$LOGLINE" >> "$LOGFILE"
	    ;;
    esac
}
#======================================================================



#======================================================================
# PARSE COMMAND LINE
#----------------------------------------------------------------------
FINDORPHANS=false
FINDRENEGADES=false
PRUNE=false

while [ "x$1" != x ] ; do
    case "$1" in
	"--orp*" ) 
            FINDORPHANS=true
	    ;;
        "--ren*" ) 
            FINDRENEGADES=true 
            ;;
        "--pru*" )
	    PRUNE=true
	    ;;
        "--qui*" )
	    LOGFILE=""
	    ;;
        "--hel*" | "-h" ) 
            usage
            exit 0
            ;;
	$(echo "$1" | grep '^\-[[:alnum:]]') )
   	    echo "$1" | grep -q o && FINDORPHANS=true
	    echo "$1" | grep -q r && FINDRENEGADES=true 
   	    echo "$1" | grep -q p && PRUNE=true
   	    echo "$1" | grep -q q && LOGFILE=""
	    if echo "${1:1}" | grep -q '[^orpq]' ; then
		echo -e "\n   Unknown option: $(echo $1 | grep '[^orpq]' | sed 's|[orpq]||g')\n"
		exit 3
	    fi
	    ;;
        * ) unset DEFAULTDIRS
	    [ -d "$1" ] && TOPDIRS[${#TOPDIRS[*]}]=$(cd "$1" ; pwd) || ERRORDIRS[${#ERRORDIRS[*]}]="$1"
            ;;
    esac
    shift
done

if ! $FINDORPHANS && ! $FINDRENEGADES ; then
    FINDORPHANS=true
    FINDRENEGADES=true
fi

[ "x$DEFAULTDIRS" != x ] && TOPDIRS=( "${DEFAULTDIRS[*]}" )

if [ "x$TOPDIRS" == x ] ; then
    echo -e "\n   No directories to search.\n"
    cleanup after
    exit 4
fi

if [ "x$ERRORDIRS" != x ] ; then
    [ ${#ERRORDIRS[*]} -eq 1 ] && echo -e "\n   Skipping nonexistent directory "${ERRORDIRS[*]}"."
    [ ${#ERRORDIRS[*]} -gt 1 ] && echo -e "\n   Skipping nonexistent directories "${ERRORDIRS[*]}"."
fi

TOPDIRS=( $(echo "${TOPDIRS[*]}" | sed 's|\([^/]\)$|\1/|') )
#======================================================================



echo ""

if $FINDORPHANS ; then
    cleanup orphans

    STARTDATE=$(date +'%Y-%m-%d %T')
    START=$(date +%s.%N)

    SKIPDIRS=( $(echo "${ORPHANSKIPDIRS[*]}" | sed 's|^|\^|' | sed 's|\([^/]\)$|\1/|') )

    if $PRUNE ; then
	BASEDIRS=( $(echo "${TOPDIRS[*]}" | grep -v "${SKIPDIRS[*]}" | sed 's|^|\^|') )
    else
	BASEDIRS=( $(find ${TOPDIRS[*]} -type d -maxdepth 1 -printf '%p/\n' | sed 's|//$|/|' | grep -v "${SKIPDIRS[*]}" | sed 's|^|\^|') )
    fi

    if [ ${#BASEDIRS[*]} -eq 0 ] ; then
        echo "   No directories to search for orphans."
    else
	tput sc ; echo -n "   ---> Setting up package structure..."

	cat /var/adm/packages/* | egrep '^[[:alnum:]]{3,4}\/' | sed 's|^|\/|' | grep "${BASEDIRS[*]}" | grep -v "${SKIPDIRS[*]}" > "$TEMPFILE"
	cat /var/adm/packages/* | egrep '^etc-incoming/' | sed 's|etc-incoming|etc|' | sed 's|^|\/|' | grep "${BASEDIRS[*]}" | grep -v "${SKIPDIRS[*]}" >> "$TEMPFILE"
        cat /var/adm/scripts/* | egrep '^config' | tr -s ' ' | cut -f2 -d' ' | egrep -v '[{(]' | sed 's|^|\/|' | grep "${BASEDIRS[*]}" | grep -v "${SKIPDIRS[*]}" >> "$TEMPFILE"

	cat "$TEMPFILE" | egrep '^*[^/]$' | sort -u > "$SLACKFILES"
	cat "$TEMPFILE" | sed 's|/[^/]\+$|/|' | sort -u > "$SLACKDIRS"

	rm "$TEMPFILE"

	tput rc ; tput el ; echo -n "   ---> Orphan search in " ; tput sc

	for BASEDIR in $(echo "${BASEDIRS[*]}") ; do
	    echo -n "${BASEDIR:1}: building search tree"

	    echo "${TOPDIRS[*]}" | grep -q "$BASEDIR" && TOPDIR=true || TOPDIR=false

	    if $TOPDIR ; then
		grep -qx "$BASEDIR" "$SLACKDIRS" && echo "${BASEDIR:1}" > "$PACKAGEDIRS" || echo "${BASEDIR:1}" > "$ORPHANDIRS"
	    else
		grep "$BASEDIR" "$SLACKDIRS" > "$PACKAGEDIRS"
		find "${BASEDIR:1}" -type d -printf '%p/\n' | sed 's|//$|/|' | grep -v "${SKIPDIRS[*]}" | sort -u > "$DIRTREE"
		comm -23 "$DIRTREE" "$PACKAGEDIRS" > "$ORPHANDIRS"
	    fi
	    
	    if [ -s "$ORPHANDIRS" ] ; then
		tput rc ; tput el ; echo -n "${BASEDIR:1}: collapsing orphaned directory tree"
		
		while [ -s "$ORPHANDIRS" ] ; do
		    DIR=$(head -1 "$ORPHANDIRS")
		    echo "$DIR" >> "$ORPHANTREE"
		    grep -v "^$DIR" "$ORPHANDIRS" > "$TEMPFILE"
                    # ^^ NOT egrep HERE; egrep CRAPS OUT ON META CHARACTERS IN DIR NAMES, LIKE IN gtk+-2.2.4
		    mv "$TEMPFILE" "$ORPHANDIRS"
		done

		rm "$ORPHANDIRS" 2> /dev/null
	    fi

	    if [ -s "$PACKAGEDIRS" ] ; then
		tput rc ; tput el ; echo -n "${BASEDIR:1}: searching for orphaned files"
		
		if $TOPDIR ; then
		    grep "${BASEDIR}[^/]\+$" "$SLACKFILES" > "$PACKAGEFILES"
		    echo "${BASEDIR:1}" > "$DIRTREE"
		else
                    grep "$BASEDIR" "$SLACKFILES" > "$PACKAGEFILES"
		    #look "${BASEDIR:1}" "$SLACKFILES" > "$PACKAGEFILES"  
                    # look SEEMS SLOWER THAN grep... ODD, IT SHOULD TAKE ADVANTAGE OF THE SORTED FILES.
		fi

		for DIR in $(comm -12 "$PACKAGEDIRS" "$DIRTREE") ; do
		    grep "^${DIR}[^/]\+$" "$PACKAGEFILES" > "$PKGFILES"
		    if [ -s "$PKGFILES" ] ; then
			find "$DIR" -type f -maxdepth 1 | sort > "$TEMPFILE"
                        comm -23 "$TEMPFILE" "$PKGFILES" >> "$ORPHANFILES"
		    else
			find "$DIR" -type f -maxdepth 1 >> "$ORPHANFILES"
		    fi
		done
	    fi

	    tput rc ; tput el
	done

	if [ -s "$ORPHANTREE" ] ; then 
            rm "$TEMPFILE" 2> /dev/null
	    echo -n "the install scripts for the $(grep -c . $ORPHANTREE) apparently orphaned directories"
	    for DIR in $(< "$ORPHANTREE") ; do
                SDIR=$(echo "$DIR" | sed 's|^/||' | sed 's|/$||')
		grep -q "$SDIR" /var/adm/scripts/* || echo "$DIR" >> "$TEMPFILE"
	    done
	    [ -s "$TEMPFILE" ] && mv "$TEMPFILE" "$ORPHANTREE" || rm "$TEMPFILE"
	    tput rc ; tput el
	fi

	if [ -s "$ORPHANFILES" ] ; then 
            egrep '.new$' "$SLACKFILES" | sed 's|\.new$||' | sort -u > $DOTNEWFILES
            sort -u "$ORPHANFILES" > "$TEMPFILE"
            mv "$TEMPFILE" "$ORPHANFILES"

	    echo -n "the install scripts for the $(grep -c . $ORPHANFILES) apparently orphaned files"
	    for FILE in $(comm -23 "$ORPHANFILES" "$DOTNEWFILES") ; do
		BASENAME="$(basename $FILE)"
                egrep -q "($BASENAME|${BASENAME}.new)" /var/adm/scripts/* && grep -q "$(dirname $FILE | sed 's|^/||')" /var/adm/scripts/* \
		    || egrep -q "($FILE|${FILE:1}.new)" /var/adm/scripts/* \
		    || echo "$FILE" >> "$TEMPFILE"
	    done
	    [ -s "$TEMPFILE" ] && mv "$TEMPFILE" "$ORPHANFILES" || rm "$TEMPFILE"

	    tput rc ; tput el

	    echo -n "taking care of a special case"
	    for FILE in $(< "$ORPHANFILES") ; do
                [ "$(echo -e $FILE)" == $(echo -e "/usr/doc/kbd-1.12/utf/â\231ªâ\231¬") ] && fgrep -q "usr/doc/kbd-1.12/utf/â\231ªâ\231¬" /var/adm/packages/kbd-* || echo "$FILE" >> "$TEMPFILE"
            done
	    [ -s "$TEMPFILE" ] && mv "$TEMPFILE" "$ORPHANFILES" || rm "$TEMPFILE"

	    tput rc ; tput el
	fi

        FINISH=$(date +%s.%N)
        ELAPSED=$(echo "scale=3; $FINISH - $START" | bc -l)

        printf "\b\b\b\b finished in %8.3f seconds.\n" "$ELAPSED"

        report file orphans "$STARTDATE" "$ELAPSED"
    fi
fi



if $FINDRENEGADES ; then
    cleanup renagades

    STARTDATE=$(date +'%Y-%m-%d %T')
    START=$(date +%s.%N)

    SKIPDIRS=( $(echo "${RENEGADESKIPDIRS[*]}" | sed 's|^|\^|' | sed 's|\([^/]\)$|\1/|') )

    if $PRUNE ; then
	BASEDIRS=( $(echo "${TOPDIRS[*]}" | grep -v "${SKIPDIRS[*]}" | sed 's|^|\^|') )
    else
	BASEDIRS=( $(find ${TOPDIRS[*]} -type d -maxdepth 1 -printf '%p/\n' | sed 's|//$|/|' | grep -v "${SKIPDIRS[*]}" | sed 's|^|\^|') )
    fi

    if [ ${#BASEDIRS[*]} -eq 0 ] ; then
        echo  "   No directories to search for renegades."
    else
        echo -n "   ---> Renegade search: "
        tput sc
        
        for PKG in /var/adm/packages/* ; do
            echo -n "${PKG##*/}"
            for FILE in $(cat "$PKG" | egrep '^[[:alnum:]]{3,4}\/[[:print:]]*[^/]$' | sed 's|^|\/|' | grep "${BASEDIRS[*]}" | grep -v "${SKIPDIRS[*]}" | sed 's|\.new$||') ; do
                [ -f "$FILE" ] || [ -f "${FILE}.new" ] || echo "${PKG##*/}: $FILE" >> "$RENEGADELIST"
            done
            
	    tput rc ; tput el
        done
        
        if [ -s "$RENEGADELIST" ] ; then
	    echo -n "checking the few special cases..."
            for FILE in $(cat "$RENEGADELIST" | cut -f2 -d' ') ; do
                [ "$FILE" == "/bin/bash2" -a -f /bin/bash ] \
                    || [ "$(echo -e $FILE)" == $(echo -e "/usr/doc/kbd-1.12/utf/â\231ªâ\231¬") -a -f $(echo -e "/usr/doc/kbd-1.12/utf/â\231ªâ\231¬") ] \
                    || echo "$FILE" >> "$TEMPFILE"
            done
	    [ -s "$TEMPFILE" ] && mv "$TEMPFILE" "$RENEGADELIST" || rm "$TEMPFILE"
	    tput rc ; tput el
        fi

        FINISH=$(date +%s.%N)
        ELAPSED=$(echo "scale=3; $FINISH - $START" | bc -l)

        printf "\b\b finished in %8.3f seconds.\n" "$ELAPSED"

        report file renegades "$STARTDATE" "$ELAPSED"
    fi
fi



cleanup after

echo -e "\n\n   SEARCH RESULTS\n"
$FINDORPHANS && report screen orphans
$FINDRENEGADES && report screen renegades
echo ""

exit 0

Cheers

thegeekster · 07-09-2004, 12:28 PM

Quote:

Originally posted by Bebo
I couldn't help myself, so I rewrote my orphans script from scratch. I hope you don't mind if I post it here....

No problem about posting it here........Although if you noticed, I made some changes in my posts in this thread so any updated code goes in the original post for the code, with an explanation in a separate post...............I figure the admin or webmaster might appreciate it a little, since this will help reduce the "clutter" (and the amount of data saved in the forum's databse) and also make it easier to find any updated code...........

One question...........How long does it take to execute your script?......................I noticed that you pushed the script's priority way back to the minimum priority..............Myself, I would just use the ampersand to run the script in the background ( <scriptname> & )

If you want to time it to the nearest second, here's a little script, called "elapsed", I whipped up for just that purpose.............To time a script just add the variable START=`date +%s at the top, and at the bottom of the script add this line (assuming you save the "elapsed" script in /usr/local/bin): /usr/local/bin/elapsed $START `date +%s` . This will give you the elapsed time to the nearest second.

Code:

#!/bin/bash
#*******************************************************************************
# Name: elapsed

if [ $# -eq 2 ]
then
  DURATION=$(( $2 - $1 ))

  hh="$(( $DURATION / 3600 ))" && hh="$(( $hh / 24 )) + $(( $hh - ( $(( $hh / 24 )) * 24 )))"
  mm="$((( $DURATION % 3600 ) / 60 ))" && [ $mm -lt 10 ] && mm=0$mm
  ss="$((( $DURATION % 3600 ) % 60 ))" && [ $ss -lt 10 ] && ss=0$ss

  echo -e "\nElapsed time is ${hh}:${mm}:$ss\n"
else
  cat << __EOF__

 Usage: $0 <start_time> <stop_time>

Shows elapsed time in 1-second intervals. The output is in the format of

  d + h:mm:ss

where "d" is the number of days elapsed, "h" is the number of hours,
"mm" is a 2-digit display of minutes. and "ss" is a 2-digit display of
seconds. 

NOTE: Start and stop times MUST be in seconds only, such as from the
      output of 'date +%s'

__EOF__

fi

NOTE: I've modified this code for the elapsed time by calculating the total number of seconds using the input from the 'date +%s' command..........Thanks to Bebo for this suggestion.........It removes the 24 hour limitation and another bug when trying to calulate the separated values of the "hh:mm:ss" input when using the input from the 'date +%T' command. ( The bug is when trying to caluclate using the digits 08 and 09 in the 2-digit format.)

Another thing is why not try to do the check on the fly, if you can.................it would save some time by not having to first create lists then having to read them.........As for the arrays, I haven't used arrays in bash scripting (yet) because I haven't found the need to.......*shrugs*

To show you what I mean, here's a sneak preview of the script I'm creating. This is a function that checks for missing files (what you call renegades, I think) on-the-fly, using 'egrep' for the filtering..................I'm still working on this but it's functional as it is:

Code:

#!/bin/sh
#*******************************************************************************
# Name: 
START1=`date +%T`

# The kernel-source and kernel-ide pkgs are excluded by default when searching
# for missing package files. To exclude other pkgs, merely add it inside the
# parentheses, separating each package name by a pipe symbol (the vertical
# symbol, | ), such as "(kernel-source|mypackage)". There must not be any spaces
# before or after the pipe symbol. If for some reason the package name contains
# a space, try escaping the space by placing a backslash _before_ the space,
# like so: "(kernel-source|my\ package)".
EXCLUDEPKG="(kernel-source|kernel-ide)"

# Excluded list of files and directories that were removed by installation plus
# the first two lines of each package list in the /var/log/packages/ directory:
EXCLUDELIST="(FILE\ LIST:|^\./$|^install|var/lib/rpm/tmp)"

# These variables are for the final list of files either missing or orphaned:
MISSING="/root/missing-files"
ORPHANS="/root/orphaned-files"

find_missing() {
  find /var/log/packages/* | egrep -v "$EXCLUDEPKG" | while read list
  do
    egrep -A5000 'FILE\ LIST:' $list | egrep -v "$EXCLUDELIST" | while read file
    do
      case $file in
        # Some manpages may have been gzipped after they were already installed:
        `echo "$file" | egrep -i '/man/'` )
          if [ ! -e `echo "${file}.gz"` -a ! -e "$file" ]
          then
            echo "`basename $list` : /$file"
          fi	;;
        # Check for "*.new" files which were renamed when installed:
        `echo "$file" | egrep -i '\.new$' | egrep -v 'bash2\.new'` )
          if [ ! -e `echo "$file" | sed 's/\.new$//'` -a ! -e "$file" ]
          then
            echo "`basename $list` : `echo \"/$file\" | sed 's/\.new$//'`"
          fi	;;
        # This is for the "glibc" package which installs the libs in a tmp
        # folder called "/lib/incoming", then moves the libs to the parent
        # directory /lib:
        `echo "$file" | egrep '/incoming/'` )
          if [ ! -e `echo "$file" | sed 's,/incoming/,/,'` -a ! -e "$file" ]
          then
            echo "`basename $list` : `echo \"$file\" | sed 's,/incoming/,/,'`"
          fi	;;
        # For the /bin/bash file which needs a separate filter all it's own:
        `echo "$file" | egrep 'bash2\.new'` )
          if [ ! -e `echo "$file" | sed 's/2\.new$//'` -a ! -e "$file" ]
          then
            echo "`basename $list` : `echo \"/$file\" | sed 's/2\.new$//'`"
          fi	;;
        # This is for a localization file installed by the "kbd" package:
        `echo "$file" | egrep '/utf/342231252342231254'` )
          if [ ! -e `echo "$file" | sed 's,/utf/342231252342231254,/utf/â™ªâ™¬,'` \
            -a ! -e "$file" ]
          then
            echo "`basename $list` : `echo \"/$file\" \
              | sed 's,/utf/342231252342231254,/utf/â™ªâ™¬,'`"
          fi	;;
        # And for the rest of the list:
        * )
          if [ ! -e "$file" ] ; then echo "`basename $list` : /$file" ; fi	;;
      esac
    done
  done
}


# Check for missing files installed by the packages:
echo "Looking for missing files..."
( cd / ; find_missing ) > $MISSING
# If the missing-files report is empty (0-byte), then...
if [ ! -s $MISSING ] ; then echo "*** No missing files. ***" > $MISSING ; fi

echo "First run: `/usr/local/bin/elapsed $START1 \`date +%T\``"

I'm using the "case" scenario because it is slightly faster than "if...then...elif", and it's easy to add or remove the filtering for special handling.........And at last count, this took a little under 50 minutes to run in the foreground while also doing other things on the computer (AMD Athlon 900 CPU with 256 MB SDRAM)......

Bebo · 07-09-2004, 09:03 PM

Hello,

I've just updated my script posted above.

Edit: Obsolete update info removed.

thegeekster,

The reason for renice'ing the script is just to stop it from taking all CPU. I don't want to ampersand it - or whatever the proper expression is - since I want to see that something is happening. Which of course also makes it slower; there are loads of echo'ing and tput'ing going on. Maybe I should give it a non-verbose option?

As for the timing... [And here I've deleted a bunch of crap about the fastness of my script that I wrote but of which I have no idea whatsoever where it came from.]

So case is faster than if? How come? BTW, if you want to calculate the elapsed time from seconds only, try date +%s

Ah yes, I forgot. The reason to chop the directory tree in small pieces and search one bit at a time is not only to have something nice (well...

) to look at. I also wanted the list to grep in to be as short as possible. It takes a very long time to do a grep AFILE -f ANOTHERFILE when the lists contain tens of thousands of words.

Well, cheers

Bebo · 07-11-2004, 05:15 PM

Here's a second update to my script above. Main points:

(1) The use of arrays is resumed. It is actually faster to search through an array than a file - if the arrays aren't too large. If I put the package files in an array instead of a file, my script will be a lot slower.

(2) The verbosity is decreased, since it gives worse performance when the info on the terminal is updated quickly.

(3) The --quick option is gone. Turns out it wasn't so quick after all. Has to do with counting the number of elements in an array methinks.

(4) New option: --prune. Only search in the given directory - do not descend in directories below.

(5) The issue with /usr/bin/[ is fixed using the -F grep option.

(6) Some logging is done, timing for instance. In nanoseconds.

[Again, I have deleted some stuff about the script's fastness. What's the point of keeping track of the performance of an erroneous script?]

This is fun

thegeekster · 07-11-2004, 09:50 PM

Quote:

Originally posted by Bebo
...This is fun

Took the words right out of my mouth.

...................Looks like I can take it slower with the script I'm working on, such as relaxing and playing a few games, etc.....

Anyway, I did another upgrade to the pkginfo script above to include any info from the installation script (doinst.sh) if one is found (not all packages have, or use, a "doinst.sh" installation script). This will be added at the bottom of the output to the screen, under the heading of "INSTALLATION FILE:" (just scroll down until you come to it)...............I thought this would be good to look at, too, for any additional information on what was done besides the adding of files from the package itself during the installation process (Remember the line that says Executing install script for <package_name>... when you install a package with the 'installpkg' command?)...........

I also did some minor code changes to the "whichpkg" script as well. There are no added features, tho', so it's not absolutely necessary to upgrade it........

Enjoy

PS: Read the section on PATTERN MATCHING in the original post above for important info on the "hybrid" pattern matching used by these scripts.

gnashley · 07-12-2004, 03:03 AM

Does one of you know the command to find a missing file in Manifest. i saw this once, but then couldn't find it. I've seen some suggestion using zless, but what I want is something like: gunzip Manifest.gz |grep "somestring" (maybe there was a 'cat' in there too), that works with it without leaving the file uncompressed.

rotvogel · 07-12-2004, 03:07 AM

Quote:

Originally posted by gnashley
Does one of you know the command to find a missing file in Manifest. i saw this once, but then couldn't find it. I've seen some suggestion using zless, but what I want is something like: gunzip Manifest.gz |grep "somestring" (maybe there was a 'cat' in there too), that works with it without leaving the file uncompressed.

Code:

zcat
zcat: compressed data not read from a terminal. Use -f to force decompression.
For help, type: zcat -h

Is that where you were looking for ?

thegeekster · 07-12-2004, 04:55 AM

Quote:

Originally posted by gnashley
Does one of you know the command to find a missing file in Manifest. i saw this once, but then couldn't find it. I've seen some suggestion using zless, but what I want is something like: gunzip Manifest.gz |grep "somestring" (maybe there was a 'cat' in there too), that works with it without leaving the file uncompressed.

Is this what you were referring to? http://www.linuxquestions.org/questi...16#post1039216

You can replace the less command with the grep command...........

gnashley · 07-12-2004, 09:48 AM

Yeh, that's a start, but I need it to get the package name too, in case the file has a name that's not part of the package name. perhaps by patching your whichpkg...
I haven't tried yours and bebos scripts yet, waiting till you both are satisfied.
But finding the package name for a missing file from the Manifest seems to be the most accurate way to track a dependency for any standard Slackware package, at least.