Problem with grep matching to end of line

icedown · 09-17-2007, 10:57 PM

Ok, I've been running linux for years and I feel like an idiot even asking this, It's probably going to be a duh answer,but anyways.

I diffed 2 files and grepped the output for just the new stuff and here was the output

Quote:

icedown@galaxy ~/data/text/stsw $ diff ls-lt ls-lt.old | grep '<'
< -rwxrwxrwx 1 gfs mtrim 2705 Sep 18 02:47 sn.0014.txt
< -rwxrwxrwx 1 gfs mtrim 2503 Sep 18 02:25 sn.0013.txt
< -rwxrwxrwx 1 gfs mtrim 2849 Sep 18 02:16 sn.0012.txt
< -rwxrwxrwx 1 gfs mtrim 1359 Sep 18 02:07 sn.0011.txt
< -rwxrwxrwx 1 gfs mtrim 1228 Sep 18 01:43 sn.0010.txt
< -rwxrwxrwx 1 gfs mtrim 1328 Sep 18 01:32 sn.0009.txt
< -rwxrwxrwx 1 gfs mtrim 1331 Sep 18 01:31 sn.0008.txt
< -rwxrwxrwx 1 gfs mtrim 2629 Sep 18 01:27 sn.0007.txt
< -rwxrwxrwx 1 gfs mtrim 1561 Sep 17 23:15 sn.0006.txt
< -rwxrwxrwx 1 gfs mtrim 1564 Sep 17 23:14 sn.0005.txt
< -rwxrwxrwx 1 gfs mtrim 1054 Sep 17 22:54 sn.0004.txt
< -rwxrwxrwx 1 gfs mtrim 1136 Sep 17 22:22 sn.0003.txt
< -rwxrwxrwx 1 gfs mtrim 1139 Sep 17 22:22 sn.0002.txt
< -rwxrwxrwx 1 gfs mtrim 1379 Sep 17 22:03 sn.0001.txt
< -rwxrwxrwx 1 gfs mtrim 1178 Sep 17 21:55 sn.0000.txt
< -rwxrwxrwx 1 gfs mtrim 1312 Sep 17 21:21 sn.0300.txt
< -rwxrwxrwx 1 gfs mtrim 1315 Sep 17 21:20 sn.0299.txt
< -rwxrwxrwx 1 gfs mtrim 1225 Sep 17 21:15 sn.0298.txt
< -rwxrwxrwx 1 gfs mtrim 1228 Sep 17 21:14 sn.0297.txt
< -rwxrwxrwx 1 gfs mtrim 1187 Sep 17 20:59 sn.0296.txt
< -rwxrwxrwx 1 gfs mtrim 1190 Sep 17 20:58 sn.0295.txt

I only want the filename part of the output so I made the command

diff ls-lt ls-lt.old | grep '<' | grep -o sn.0[0-3][0-9][0-9].txt

but nothing came out so I played with it for a little while and when i ran this

diff ls-lt ls-lt.old | grep '<' | grep -o sn.0[0-3][0-9][0-9].txt

i got the "sn.0???.tx" output, but i can't get that last "t" out of it? what am I doing wrong, I can just add it to the end of every line, that's not a problem, but I'm wondering what's going on here and how to deal with it in case i have something a little less constant in the future.

syg00 · 09-17-2007, 11:09 PM

Looks like it should work - maybe try quotes on that as well.

icedown · 09-17-2007, 11:21 PM

Quotes did it. Don't know why I didn't try that... guess i can put my dunce hat on for the night

diff ls-lt ls-lt.old | grep '<' | grep -o 'sn.0[0-3][0-9][0-9].txt'

Thank you, I was about to pull my hair out. I am still curious as to why that is needed. Is there something special about the end of the line or could it be a bug?

jschiwal · 09-18-2007, 12:50 AM

You could use "cut" or "awk" to cut out the column you want.

diff ls-lt ls-lt.old | grep '<' | cut -d' ' -f5

diff ls-lt ls-lt.old | awk '/</{ print $5 }'

icedown · 09-18-2007, 01:25 AM

ok, i'll keep that in mind. I'm writing some scripts that are going to be dealing with more complex filenames and such. I was just curious if there was something that grep was seeing that I don't know about and may become an issue later with other programs like sed and others that use regular expressions

syg00 · 09-18-2007, 02:17 AM

Must be a problem at your machine - I cut some of your data, and the first regex (no quotes) worked fine.

secretlydead · 09-20-2007, 04:26 AM

How do I start grep at line, say, 200?

(My output is too large and produces a segmentation fault, so i'm going to have to grep each 100 or so lines at a time.)

chrism01 · 09-20-2007, 08:14 PM

Maybe wrap it in a loop thus:

Code:

for rec in `cat yourfile`
do
    grep stuff here $rec
done

which will process it rec by rec, slower, but file size should not be an issue.

syg00 · 09-20-2007, 09:24 PM

Mmmmm - don't know about that; lots of blanks in there to trip over the default IFS.
It were me doing this, at about this point I'd be starting to think perl ....

farkus888 · 09-20-2007, 10:38 PM

Quote:

Originally Posted by secretlydead

How do I start grep at line, say, 200?

(My output is too large and produces a segmentation fault, so i'm going to have to grep each 100 or so lines at a time.)

did you have the same problem with this awk version?

Code:

diff ls-lt ls-lt.old | awk '/</{ print $5 }'

it seems this could be exactly what you want. quick and simple way to do it.

chrism01 · 09-21-2007, 02:45 AM

You can set IFS to newline only easy enough.
Perl would be faster though if the file to process is that big, just seemed a bit excessive if he just wants the last field of each rec.
Also depends if this is just a one-off requirement or if its going to be run multiple times.

ghostdog74 · 09-21-2007, 03:36 AM

Code:

awk '/^</{for (i=1;i<=9;i++ ) $i=""}
         { gsub(" +","",$0) ; print}' "file"

jschiwal · 09-23-2007, 05:43 AM

If you want to use a range of lines, in sed you can start a sed command with a range like,
2000,2500 or use a regular expression like /\[General\]/,/$^/.
You can also use brackets to try to find a match within a range.
Also, using sed, you can add a quit command if you have found a match. That is commonly done to avoid reading all of the lines in a large file.

Code:

2000,2500{
    /pattern/<command>
         }
2501q

Using grep, look for the option to return a single result. That will abort after finding a match, and won't continue reading till the end of the file looking for another match.

The cut command would be ideal for cutting out a certain field. That is exactly what the command was written for. You could use head to select the top N lines and pipe the output to cut.

If you want to use a certain range of lines, you can use sed to filter through only those lines and pipe the output to the cut command:

Code:

diff ls-lt ls-lt.old | sed -n "${start},${end}p;$((${end}+1))q | grep '<' | cut -d' ' -f5