Need a little SED help...

jong357 · 07-27-2006, 01:12 PM

I've almost got myself where I need to be but my command is too specific. Here's what I'm dealing with.

Code:

for tarball in $(cat pkglist)
do
  srcdir=$(echo $tarball | sed 's/.tar.bz2//')
  modpackagename=$(echo $srcdir | sed 's/\-[0-9]\.[0-9]\.[0-9]//')
  echo $modpackagename
done

$modpackagename is what's giving me a headache. I'm trying to break down the entire tarball name to just the package name. It works great as long as the version number is -1.2.3 or -4.2.5 ect... But when I hit -4.0 or -1.13.2 it skips over it, exactally as I've told it to.

As soon as it finds a hypen followed imediately by any number, I need it to ditch the hypen, number and EVERYTHING after that.

Learning SED can be pretty difficult at times...

I'm making my own Xorg-7.1 build script if you haven't figured it out..

Not only do I need indivdual package names for the slack-desc's but not all packages build with the generic commands so I need specific build functions for certain packages and I DO NOT want to hard code version numbers.

Thanks for any help guys (and gals)...

spirit receiver · 07-27-2006, 01:33 PM

Try this:

Code:

sed 's/-[[:digit:].]\+$//'

jong357 · 07-27-2006, 01:41 PM

Nice....

Thanks much!

jong357 · 07-29-2006, 08:59 PM

Alright... Stuck again only this one seems a little more tricky.

I swear this is the last one.

I have a text file, well, this text file.

Code:

applewmproto-1.0.3.tar.bz2
bigreqsproto-1.0.2.tar.bz2
compositeproto-0.2.2.tar.bz2
compositeproto-0.3.1.tar.bz2
compositeproto-0.3.tar.bz2
damageproto-1.0.3.tar.bz2
dmxproto-2.2.2.tar.bz2
evieext-1.0.2.tar.bz2
fixesproto-3.0.2.tar.bz2
fixesproto-4.0.tar.bz2
fontcacheproto-0.1.2.tar.bz2
fontsproto-2.0.2.tar.bz2
glproto-1.4.3.tar.bz2
glproto-1.4.4.tar.bz2
glproto-1.4.5.tar.bz2
glproto-1.4.6.tar.bz2
glproto-1.4.7.tar.bz2
inputproto-1.3.2.tar.bz2
kbproto-1.0.2.tar.bz2
kbproto-1.0.3.tar.bz2
printproto-1.0.3.tar.bz2
randrproto-1.1.2.tar.bz2
recordproto-1.13.2.tar.bz2
renderproto-0.9.2.tar.bz2
resourceproto-1.0.2.tar.bz2
scrnsaverproto-1.0.2.tar.bz2
scrnsaverproto-1.1.0.tar.bz2
trapproto-3.4.3.tar.bz2
videoproto-2.2.2.tar.bz2
windowswmproto-1.0.3.tar.bz2
xcmiscproto-1.1.2.tar.bz2
xextproto-7.0.2.tar.bz2
xf86bigfontproto-1.1.2.tar.bz2
xf86dgaproto-2.0.2.tar.bz2
xf86driproto-2.0.3.tar.bz2
xf86miscproto-0.9.2.tar.bz2
xf86rushproto-1.1.2.tar.bz2
xf86vidmodeproto-2.2.2.tar.bz2
xineramaproto-1.1.2.tar.bz2
xproto-7.0.4.tar.bz2
xproto-7.0.5.tar.bz2
xproto-7.0.6.tar.bz2
xproto-7.0.7.tar.bz2
xproxymanagementprotocol-1.0.2.tar.bz2

How can I ditch the repeating lines(package names) but only keep the highest version number line?

It would be significantly easier I think if the latest version number was always the last occurance but look at compositeproto....

I'll donate 10 dollars to the charity of your choice if an answer is given... An answer that doesn't involve manually editing the file that is...

Thanks all for your time...

konsolebox · 07-29-2006, 10:49 PM

hello jong357

perhaps this script can make you donate 10 dollars to charity

the usage of this script is simple. just do 'freshpick.sh list'.

freshpick.sh

Code:

#!/bin/bash

# freshpick.sh (c) 2006 konsolebox
# chooses the newest versions of packages from a list
# version: 1.0
# license: GPL

DEBUG=

debug() { [ "$DEBUG" ] && echo "$@"; }
error() { echo "error: $@"; exit; }

usege() {
	echo "usage: freshpick.sh list"
	exit
}

compareversions() {
	for ((p=1; p<=10; p++)); do
		a=$(echo "$1" | cut -f "$p" -d '.')
		b=$(echo "$2" | cut -f "$p" -d '.')
		debug "comparing $a to $b"

		if [ -z "$a" ] && [ -z "$b" ]; then
			debug "compareversions exit no 1"
			return 1
		elif [ -z "$b" ]; then
			debug "compareversions exit no 2"
			return 1
		elif [ -z "$a" ]; then
			debug "compareversions exit no 3"
			return 0
		fi

		if [[ b -gt a ]]; then
			debug "compareversions exit no 4"
			return 0
		elif [[ b -lt a ]]; then
			debug "compareversions exit no 5"
			return 1
		fi

		# a still equals to b so continue
	done

	error "versions $a and $b too long"
}

finalize() {
	# you an modify this function if you like
	echo "$@"
}


#### start ####

FILENAME="$1"
if [ -z "$FILENAME" ]; then
	usege
elif [ ! -e "$FILENAME" ]; then
	error "$FILENAME" not found
fi

IFS=$'\n'
for fn in $(cat "${FILENAME}" | sort); do
	NEWPACKAGE="${fn/-*}"
	NEWVERSION="$(echo "$fn" | sed s/".*\-\([0-9.]\+[0-9]\).*"/"\1"/)"	# better than 'grep -o ..'
	debug "$NEWPACKAGE" "$NEWVERSION"
	if [ "${NEWPACKAGE}" != "${CURRENTPACKAGE}" ]; then
		[ "${CURRENTPACKAGE}" ] && \
			finalize "${CURRENTPACKAGE}-${NEWESTVERSION}.${CURRENTSUFFIX}"
		CURRENTPACKAGE="${NEWPACKAGE}"
		NEWESTVERSION="${NEWVERSION}"
		CURRENTSUFFIX="$(echo "$fn" | sed s/".*${NEWVERSION}."//)"
	else
		compareversions "${NEWESTVERSION}" "${NEWVERSION}" && \
			NEWESTVERSION="${NEWVERSION}"
	fi
done

IFS=$' :\t\n'

this script is not yet optimized though. but i hope it serves its purpose.

regards

jong357 · 07-30-2006, 12:19 AM

Wow... That's slick... No kiddin. Thanks!

Here's a snip of what I'm up to if anyone is interested. As stated it's for my xorg-7.1 .SlackBuilds.. This will set me up nicely for my for/in/do loop.

Code:

wget http://xorg.freedesktop.org/releases/individual/proto/
grep -o 'f=".*bz2' index.html > new 
sed 's/f="//' new > new2
sed 's/".*//' new2 > new3
sh freshpick.sh new3 > pkglist
rm -f index.html new new2 new3

It's snipping off the last package tho in pkglist.. I'll investigate tommorrow. Getting late. Thanks again konsolebox.. If that's you on kerneltrap, throw up a paypal link..

konsolebox · 07-30-2006, 04:42 AM

Quote:

Originally Posted by jong357

Thanks again konsolebox.. If that's you on kerneltrap, throw up a paypal link..

LOL!

Anyway pls don't forget to tell me if you find an error and made a fix ok. Someday i'll post this damn scripts in a blog site.

........ Nah.

spirit receiver · 07-30-2006, 07:53 AM

There's something wrong with konsolebox' script. I used it on the following data:

Code:

recordproto-1.13.2.tar.bz2
recordproto-1.9.2.tar.bz2
glproto-1.4.5.tar.bz2
glproto-1.4.6.tar.bz2
randrproto-1.1.2.tar.bz2
randrproto-1.1.tar.bz2

which returned

Code:

glproto-1.4.6.tar.bz2
randrproto-1.1.2.tar.bz2

I also gave it a try, but it's still awfully slow. A Perl script would certainly do much better.

Code:

#!/bin/bash

DATAFILE=kram

function grab_latest() {
    local LATEST              # will eventually contain the latest archive
    local -a LATEST_VER=( 0 ) # and its version, fields separated in an array
    while read CURRENT
    do

      # split version information into an array
      local -a VER=( $(echo $CURRENT | 
	  sed -r 's/^.*-([[:digit:].]+)\.tar\.bz2/\1/;s/\./ /g') )

      # iterate over fields of that array
      for (( i=0; i < ${#VER[*]}; i++ ))
      do
	# compare current field and replace LATEST if appropriate
	if (( VER[i] != LATEST_VER[i] ))
	then
	    if (( VER[i] > LATEST_VER[i] ))
	    then
		LATEST=$CURRENT
		LATEST_VER=( ${VER[*]} )
	    fi
	    break
	fi
      done
    done
    echo $LATEST
}

# file handle 3 will deliver the package names (without version info) that we'll process
exec 3< <( sed  -n 's/-[[:digit:].]\+tar\.bz2$//p' $DATAFILE | sort | uniq )

# now look for the latest version of each package
while read <&3 
do
  grep "^$REPLY" $DATAFILE | grab_latest
done

konsolebox · 07-30-2006, 08:05 AM

Ok tomorrow i'll try to fix the script. I just need some sleep now.

jong357 · 07-30-2006, 10:12 AM

Man, I really aught to brush up on my scripting. Must be nice to come up with stuff like this when the need arises...

Everytime I start messing around with sed I realize you can pretty much do anything with it.

Yea, it did snip recordproto on that short list. It snipped xproxymanagementprotocol on my list as well. Haven't tried it on anything else yet. Thanks go out to both of you... REALLY. You have no idea how much I appreciate it.

Since you guys are having SOOOOOOOOOOO much fun doing this,

, I know your just shaking your head at the way I'm retrieving a file list from freedesktop.org.... Your just itching to do a quick rewrite of it, I'm sure...

Here's my finished script for xorg-proto if it will be of any use to someone. I'm sure I'll find a few things to change over the next couple days. It also serves as a nice template for the other Xorg module bases... I typically don't like to use so many variables but wanted it adaptable for the other modules with minimum changes. It's all downhill from here. I started this because:

1. I'm not sure when Pat will get around to doing it. He's a little busy with prepping for 11.0
2. The existing 'how-to's' here on LQ left a hell of alot to be desired. Posting steps 1-35 is NOT a solution IMO...
3. I like to make my own build scripts/packages anyway...

Code:

#!/bin/sh

CWD=`pwd`
TMP=/tmp
PKGNAME=xorg-proto
VERSION=7.1
PKGLST=$CWD/sources/$PKGNAME.lst
PKGDIR=$TMP/Xorg${VERSION}/packages/modules
PKG=$TMP/Xorg${VERSION}/packages/$PKGNAME
SRCPOOL=$TMP/Xorg${VERSION}/source/$PKGNAME
FETCH=http://xorg.freedesktop.org/releases/individual/proto/
ARCH=noarch
BUILD=1
# FHS with /usr or should we go /usr/X11R7 ? Not sure yet...
XORG_PREFIX=/usr
XORG_CONFIG="./configure --prefix=$XORG_PREFIX --sysconfdir=/etc \
    --localstatedir=/var --datadir=$XORG_PREFIX/share"
# Do we need to sequentially install the modular packages?
INSTALLMOD=no

if [ ! -s $PKGLST ]; then
   echo
   echo "Retrieving most current package list..."
   mkdir -p $CWD/sources
   wget $FETCH
   grep -o 'f=".*bz2' index.html > new 
   sed 's/f="//' new > new2
   sed 's/".*//' new2 > new3
   # FIXME - The last package isn't making it into $PKGLST
   sh $CWD/freshpick.sh new3 > $PKGLST
   rm -f index.html new new2 new3
fi
if [ ! -a $CWD/sources/$PKGNAME/* ]; then
   echo
   echo "Fetching source tarballs..."
   mkdir -p $CWD/sources/$PKGNAME
   ( cd $CWD/sources/$PKGNAME
     wget -B $FETCH -i $PKGLST )
fi

rm -rf $PKG
rm -rf $PKGDIR/$PKGNAME
rm -rf $SRCPOOL
mkdir -p $PKG/install
mkdir -p $PKGDIR/$PKGNAME
mkdir -p $SRCPOOL

# Build Function:
build_all()
{ :
for tarball in $(cat $PKGLST)
do
  cd $SRCPOOL
  srcdir=$(echo $tarball | sed 's/.tar.bz2//')
  modname=$(echo $srcdir | sed 's/-[[:digit:].]\+$//')
  modpackage=$PKGDIR/$PKGNAME/package-$srcdir
  echo "######################################################################"
  tar -xvf $CWD/sources/$PKGNAME/$tarball
  cd $srcdir
  $XORG_CONFIG
  make
  make install DESTDIR=$PKG
  if [ "$INSTALLMOD" = "yes" ]; then
     make install DESTDIR=$modpackage
     cd $modpackage
     makepkg -l y -c n $PKGDIR/$PKGNAME/$srcdir-$ARCH-${BUILD}_xorgmod.tgz > /dev/null 2>&1
     installpkg $PKGDIR/$PKGNAME/$srcdir-$ARCH-${BUILD}_xorgmod.tgz > /dev/null 2>&1
     /sbin/ldconfig
  fi
done
}

# Build:
echo
echo "'tail -f $SRCPOOL/$PKGNAME.log' to watch the output."
echo "Also, keep an eye on $SRCPOOL/$PKGNAME.err"
echo
echo "Please wait while $PKGNAME-$VERSION is building..."
{ build_all 3>&1 1>&2 2>&3 | tee "$SRCPOOL/$PKGNAME.err" ;} &>"$SRCPOOL/$PKGNAME.log"

# Compress and strip:
# FIXME - Won't work on deep man dir
gzip -9 $PKG/usr/man/man?/* > /dev/null 2>&1
( cd $PKG
  find . | xargs file | grep "executable" | grep ELF | cut -f 1 -d : | xargs strip --strip-unneeded 2> /dev/null
  find . | xargs file | grep "shared object" | grep ELF | cut -f 1 -d : | xargs strip --strip-unneeded 2> /dev/null )

# Make the package description:
cat << EOF > $PKG/install/slack-desc
# HOW TO EDIT THIS FILE:
# The "handy ruler" below makes it easier to edit a package description. Line
# up the first '|' above the ':' following the base package name, and the '|'
# on the right side marks the last column you can put a character in. You must
# make exactly 11 lines for the formatting to be correct. It's also
# customary to leave one space after the ':'.

          |-----handy-ruler----------------------------------------------------|
xorg-proto: xorg-proto (Protocol Headers)
xorg-proto:
xorg-proto: The Xorg protocol headers provide the header files and pkgconfig
xorg-proto: files required to build the X window system, and to allow other
xorg-proto: applications to build against it as well.
xorg-proto:
xorg-proto:
xorg-proto:
xorg-proto:
xorg-proto:
xorg-proto:
EOF

# Ditch the spam:
if [ "$INSTALLMOD" = "yes" ]; then
   removepkg /var/log/packages/*xorgmod > /dev/null 2>&1
fi

# Build the package:
cd $PKG
makepkg -l y -c n $TMP/$PKGNAME-$VERSION-$ARCH-$BUILD.tgz

Not much to it! I was really iritated by Xorg leaving the traditional imake way of building and got really nervous about hundreds of potential packages but luckily, most of them can be used in a loop. Nice.... It would also be extremely silly to build them individually. I was going to make a monolithic package but decided the script would get too mesy. Instead I opted for building each module base into one package. That will make about 8 packages or so excluding mesa and a couple others. Not bad...

There's also a dependency order with the packages on some of the other module direcotries. Haven't quite figured out how to handle that one yet. I may have to abandon my $PKGLST current idea and just have a static $PKGLST in place. Man, what a pain...

jong357 · 07-30-2006, 11:36 AM

Quote:

Originally Posted by spirit receiver

...snip

Code:

line 34: syntax error near unexpected token `<'
line 34: `exec 3< <( sed  -n 's/-[[:digit:].]\+tar\.bz2$//p' $DATAFILE | sort | uniq )'

Hmm.... Any idea? I replaced the 2nd < with a $ and it just spits out the $PKGNAME's but without the $VERSION numbers and also says:

xproxymanagementprotocol: File name too long

A little over my head...

And becoming problematic the more I think about it. If these 2 scripts would function properly AND I figured out the build order problem, it'd be a viable solution. I'd still like to have one or both of those scripts working tho. I can think of several things to use them for at the moment besides these build scripts..

Thanks again guys and if I get on your nerves just forget about it. I'll work it out somehow... It's been a good learning expierence for me thus far.

spirit receiver · 07-30-2006, 12:49 PM

That syntax is correct, it's called process substitution. But I'm probably relying on GNU Bash a lot in that script. What does "bash --version" give?

This redirection thingy isn't necessary here, you can also use a pipe, but I expect that you'll run into trouble somewhere else. However, you can give it a try:

Code:

#!/bin/bash

DATAFILE=filename


function grab_latest() {
    local LATEST              # will eventually contain the latest archive
    local -a LATEST_VER=( 0 ) # and its version, fields separated in an array
    while read CURRENT
    do

      # split version information into an array
      local -a VER=( $(echo $CURRENT | 
	  sed -r 's/^.*-([[:digit:].]+)\.tar\.bz2/\1/;s/\./ /g') )

      # iterate over fields of that array
      for (( i=0; i < ${#VER[*]}; i++ ))
      do
	# compare current field and replace LATEST if appropriate
	if (( VER[i] != LATEST_VER[i] ))
	then
	    if (( VER[i] > LATEST_VER[i] ))
	    then
		LATEST=$CURRENT
		LATEST_VER=( ${VER[*]} )
	    fi
	    break
	fi
      done
    done
    echo $LATEST
}

# pipe the package names (without version info) to the loop
( sed -n 's/-[[:digit:].]\+tar\.bz2$//p' $DATAFILE | sort | uniq )|

# now look for the latest version of each package
while read 
do
  grep "^$REPLY" $DATAFILE | grab_latest
done

jong357 · 07-30-2006, 01:22 PM

[root@jaguar ~/Desktop/xorg-7.1] bash --version
GNU bash, version 3.1.14(1)-release (powerpc-unknown-linux-gnu)
Copyright (C) 2005 Free Software Foundation, Inc.

Eh... Sorry. Was calling it with sh... REALLY bad habit. It works..

Thanks again for all your help. I think if I have a buildorder.txt with the correct order of package names, redirect the output of your script to pkglst.txt, and then come up with something to compare the 2 and make a new pkglst.txt that's in the correct order, I should be golden... Now to figure that one out..

Thanks again! Fun stuff.

spirit receiver · 07-30-2006, 04:23 PM

Quote:

Originally Posted by jong357

compare the 2 and make a new pkglst.txt that's in the correct order

That should be pretty easy. Read a line from buildorder.txt and grep pkglst.txt for the result. But you could also feed the "while read <&3" loop of my script with the lines of buildorder.txt which will give the right order immediately.

jong357 · 07-30-2006, 05:14 PM

I'm a little toasty from working on this and other stuff this weekend. I'll jump on it in a few days when I'm thinking more clear. I'm not sure how just grepping for a pkgname is going to rearange the output from your script to a new order listing... Both .txt files will have the same pkgnames but only one will have version numbers (the output from your script)...

Like I say, I'll have to have a look later. Feeling the screen burn right now...

Wouldn't using the last loop of that script also make it try to determine version # thru grab_latest? At any rate, thanks again... Got a good jump on what I was trying to do.