extract lines from log file using different parameters

veda92 · 07-03-2013, 04:36 AM

Hi,
I am an intern, very new to linux and told to write a program where certain parameters entered, extracts information from different files
In /var/log there are router,route,routers and route.1,.2,.3 similarly with others
the file of the following pattern:

Jul 3 14.38.53 10.85.521 1230: Jul 3 14.32.03 %link updown:interface has turned down
Jul 3 14.35.65 25.65.897 5698: jul 3 08.26.35 %duplex mismatch tunnel 10

Now column 1 refers to month then date,time ip address,then the time and date when the kind of errors had occured.
If one were to give i/p as some ip address or error "duplex mismatch" ,o/p should have all the lines containing the ip address or error in a different file.
I don't know how to go about it should I use awk to convert them to fields and use field variable in grep? If not the script atleast someone could please suggest how do i go about it.

chrism01 · 07-03-2013, 05:42 AM

Here's a good cli tutorial http://rute.2038bug.com/index.html.gz.

Are you sure about what you've been told '10.85.521 1230' doesn't look like an IP address.

In general, grep would probably do eg

Code:

grep -E 'pattern1|pattern2' filename

David the H. · 07-03-2013, 04:41 PM

This sounds a bit homework-like to me, so I'd hesitate to give any exact solutions.

In general though, grep is a fairly simple program that pulls whole lines that contain a given regular expressions pattern somewhere in them. It's very fast and efficient, but limited to whatever regex you can devise. If it's possible to extract the lines you want with grep, then consider using it first.

awk is a much more powerful and feature-rich text processing scripting language. If you need to do fine-tuned matches, including matching sub-fields and making numerical comparisons and alterations, and re-formatting the output, then that's the one you want to use.

veda92 · 07-04-2013, 12:12 AM

Quote:

Originally Posted by chrism01

Here's a good cli tutorial http://rute.2038bug.com/index.html.gz.

Are you sure about what you've been told '10.85.521 1230' doesn't look like an IP address.

In general, grep would probably do eg

Code:

grep -E 'pattern1|pattern2' filename

thanks for the link its was informative.I can't use pattern its variable also the filename is variable here and the result ought to be printed on another page

descendant_command · 07-04-2013, 12:30 AM

Code:

grep -E '$1|$2' $3 > $4

David the H. · 07-05-2013, 12:14 PM

Quote:

Originally Posted by veda92

I can't use pattern its variable also the filename is variable here and the result ought to be printed on another page

You can, you just have to get the quotes right. Variables will only expand inside double quote marks. Single quotes make everything literal.

Code:

var1=foo
var2=bar
var3=/path/to/inputfile.txt
var4=/path/to/outputfile.txt

grep -E "$var1|$var2" "$var3" >"$var4"

It's vital in scripting to understand how the shell handles arguments and whitespace. See these links:

http://mywiki.wooledge.org/Arguments
http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes

konsolebox · 07-07-2013, 05:23 AM

Fgrep would be helpful. And you could use this pattern to extract lines from multiple files along with multiple keywords:

Code:

fgrep -e keyword1 -e keyword2 -e keyword3 ... -- file1 file2 file3

In bash you could make use of read -a to get keywords:

Code:

#!/bin/bash
read -a KEYWORDS

Then process the contents of KEYWORDS to be usable by fgrep:

Code:

PATTERNS=()
for A in "${KEYWORDS[@]}"; do
    PATTERNS=("${PATTERNS[@]}" -e "$A")  ## PATTERNS+=(-e "$A") for newer shells but with less compatibility.
done

From there you can now call fgrep.

Code:

fgrep -H "${PATTERNS[@]}" -- "$@"  ## "$@" should hold the files passed as an argument to the script; -H makes sure that the filename is shown even with single files

And the complete form of it would be:

Code:

#!/bin/bash
read -a KEYWORDS
PATTERNS=()
for A in "${KEYWORDS[@]}"; do
    PATTERNS=("${PATTERNS[@]}" -e "$A")
done
fgrep -H "${PATTERNS[@]}" -- "$@"

If you want, you could also let your script accept both keywords and files as arguments but are separated with -- keyword.

Code:

#!/bin/bash
PATTERNS=()
FILES=()
while [[ $# -gt 0 ]]; do
    case "$1" in
    --)
        shift
        while [[ $# -gt 0 ]]; do
            FILES=("${FILES[@]}" "$1")
            shift
        done
        ;;
    *)
        PATTERNS=("${PATTERNS[@]}" -e "$1")
        ;;
    esac
    shift
done

if [[ ${#FILES[@]} -eq 0 ]]; then
    echo "No file was entered."
    exit 1
elif [[ ${#PATTERNS[@]} -eq 0 ]]; then
    echo "No pattern was entered."
    exit 1
fi

fgrep -H "${PATTERNS[@]}" -- "${FILES[@]}"

And run the command like:

Code:

[bash] script[.sh] abc def ghi jkl -- /var/log/x /var/log/y

Lastly in a more advanced manner, you could use exec to call fgrep so that you could just replace the shell's process with the command.

Code:

exec fgrep -H "${PATTERNS[@]}" -- "${FILES[@]}"

With this we assume that you wouldn't call the script with . or source which is never likely is it?

If you find the output with long lines hard to read, you could use less -S:

Code:

... | less -S

veda92 · 07-07-2013, 09:36 PM

thanks it worked

veda92 · 07-07-2013, 09:39 PM

thanks kconsole it worked and helped another problem as well

konsolebox · 07-07-2013, 11:44 PM

You're welcome. There is also one thing. You can add -i as an option to fgrep to ignare case when searching with keywords.

Code:

fgrep -H -i "${PATTERNS[@]}" -- "${FILES[@]}"

And you can also make your script parse it to make it optional.

Code:

#!/bin/bash
PATTERNS=()
FILES=()
DASHI=()
while [[ $# -gt 0 ]]; do
    case "$1" in
    --)
        shift
        while [[ $# -gt 0 ]]; do
            FILES=("${FILES[@]}" "$1")
            shift
        done
        ;;
    -i)
        DASHI=("-i")
        ;;
    *)
        PATTERNS=("${PATTERNS[@]}" -e "$1")
        ;;
    esac
    shift
done

if [[ ${#FILES[@]} -eq 0 ]]; then
    echo "No file was entered."
    exit 1
elif [[ ${#PATTERNS[@]} -eq 0 ]]; then
    echo "No pattern was entered."
    exit 1
fi

fgrep -H "${DASHI[@]}" "${PATTERNS[@]}" -- "${FILES[@]}"

So you could use it like:

Code:

[bash] script[.sh] -i abc def ghi jkl -- /var/log/x /var/log/y

---- EDIT ----

Come to think of it we no longer have to loop through with the files.

Code:

#!/bin/bash
PATTERNS=()
FILES=()
DASHI=()
while [[ $# -gt 0 ]]; do
    case "$1" in
    --)
        FILES=("${@:2}")
        break
        ;;
    -i)
        DASHI=("-i")
        ;;
    *)
        PATTERNS=("${PATTERNS[@]}" -e "$1")
        ;;
    esac
    shift
done

if [[ ${#FILES[@]} -eq 0 ]]; then
    echo "No file was entered."
    exit 1
elif [[ ${#PATTERNS[@]} -eq 0 ]]; then
    echo "No pattern was entered."
    exit 1
fi

fgrep -H "${DASHI[@]}" "${PATTERNS[@]}" -- "${FILES[@]}"

Or even yet no longer use the FILES variable at all and just use "$@", but now it's just a preference.