find the number in the file

byran cheung · 10-30-2013, 05:24 AM

In my system , there are text files will be generated monthly , the file name begins with xxx , then year , month ( for example xxxxx201310.txt means Oct 2013 )

I have below command to count how many abc in the month , but it only count the number in this month .

NUMBER=$(cat xxxxx201310.txt |grep -c -s "abc" )

Could advise if I would like to have the following report , it shows the no. of abc for each month , and the total no. of abc , how to make it ? very thanks

no of abc in Oct 2013 = 111
no of abc in Nov 2013 = 222
no of abc in Dec 2013 = 333
"
"
"
Total no of abc = 999

pan64 · 10-30-2013, 05:31 AM

first, you do not need to use cat, grep -c -s abc xxxxx201310.txt is enough
next, grep can handle more files, just try: grep -c -s abc xxxxx*.txt
and finally you need a small script to sum up the numbers and print a formatted output.
Can you do that yourself?

danielbmartin · 10-30-2013, 10:55 AM

This will get you started ...

With this InFile ...

Code:

abc201309.txt
def201309.txt
pqr201311.txt
abc201311.txt
abc201309.txt
ghi201310.txt
ghi201310.txt
abc201310.txt
abc201311.txt
mno201311.txt
def201311.txt
abc201311.txt
jkl201310.txt
abc201310.txt
abc201311.txt

... this code ...

Code:

awk '{print substr($0,4,6)substr($0,1,3)}'  $InFile  \
|sort  \
|sed '$ a\**End Of File**'  \
|awk 'BEGIN{getline; prev=$0}
    {if (prev==$0) c++
   else {print "In year",substr(prev,1,4),
               "month",  substr(prev,5,2),
               "string", substr(prev,7,3),
               "was seen",c,"times"; prev=$0; c=1}}' >$OutFile

... produced this OutFile ...

Code:

In year 2013 month 09 string abc was seen 1 times
In year 2013 month 09 string def was seen 1 times
In year 2013 month 10 string abc was seen 2 times
In year 2013 month 10 string ghi was seen 2 times
In year 2013 month 10 string jkl was seen 1 times
In year 2013 month 11 string abc was seen 4 times
In year 2013 month 11 string def was seen 1 times
In year 2013 month 11 string mno was seen 1 times
In year 2013 month 11 string pqr was seen 1 times

Note: I tried (but failed) to do this with an awk one-liner, using an array and asorti. I'd love to see how this could be done. Gurus?

Daniel B. Martin

byran cheung · 10-30-2013, 10:48 PM

hi Daniel ,

thanks reply , what I would like to do is to find how many times the string "abc" occur , but not abc , def , ghi ...

thanks .

Firerat · 10-31-2013, 01:36 AM

assuming files are 'fixed' like xxxx201310.txt

Code:

MyString="abc"
for File in *.txt;do
    echo "No. of \"${MyString}\" in $(date "+%b %Y" --date="${File:4:6}01") = $(grep -c "${MyString}" "$File")"
done

NOTE: here grep is only counting lines
since you don't provide sample, I have no idea if that is what you want/need

not great, but fudged an awk to count each occurrence

Code:

MyString="abc"
for File in *.txt;do
    echo "No. of \"${MyString}\" in $(date "+%b %Y" --date="${File:4:6}01") = $(awk 'BEGIN{C=0}{for (i=1;i<=NF;i++) if ( $i ~ '${MyString}' ){C++}} END{print C}' "$File")"
done

be careful

"the" and "then" are two different words, but both contain "the"

byran cheung · 10-31-2013, 03:03 AM

thx Firerat ,

It works for count the number , could advise how to calculate the total number for all months ?

thanks

pan64 · 10-31-2013, 03:09 AM

Firerat, why do you use an awk script instead of grep -c "pattern" filename?
byran cheung, what is the problem with my tip?

Code:

for File in *.txt;do
C=$(grep -c "abc" $File)
SUM=$((SUM+C))
echo "$File: $C"
done
echo $SUM

(or something like this will do the job)
(not tested)

Firerat · 10-31-2013, 03:34 AM

grep -c only returns the number of lines with the pattern match
the awk I gave counts all occurrences of the pattern

thought I had explained that in my post

pan64 · 10-31-2013, 03:38 AM

based on the original post grep is sufficient, but I'm not really sure about that

Firerat · 10-31-2013, 03:39 AM

Quote:

Originally Posted by byran cheung

thx Firerat ,

It works for count the number , could advise how to calculate the total number for all months ?

thanks

adapt a little

Code:

Total=0
MyString="abc"
for File in *.txt;do
    Count=$(awk 'BEGIN{C=0}{for (i=1;i<=NF;i++) if ( $i ~ '${MyString}' ){C++}} END{print C}' "$File")
    echo "No. of \"${MyString}\" in $(date "+%b %Y" --date="${File:4:6}01") = $Count"
    Total=$(( $Count + $Total ))
done
echo "Total No. of \"${MyString}\" = $Total"

Firerat · 10-31-2013, 03:41 AM

Quote:

Originally Posted by pan64

based on the original post grep is sufficient, but I'm not really sure about that

hence :

Quote:

Originally Posted by Firerat

...
NOTE: here grep is only counting lines
since you don't provide sample, I have no idea if that is what you want/need

not great, but fudged an awk to count each occurrence
...

danielbmartin · 10-31-2013, 09:29 AM

With this InFile ...

Code:

abc201309.txt
def201309.txt
pqr201311.txt
abc201311.txt
abc201309.txt
ghi201310.txt
ghi201310.txt
abc201310.txt
abc201311.txt
mno201311.txt
def201311.txt
abc201311.txt
jkl201310.txt
abc201310.txt
abc201311.txt

... this awk ...

Code:

awk '{if (substr($0,1,3)=="abc") c[substr($0,4,6)]++}
   END{n=asorti(c,d); for (j=1;j<=n;j++)
       print "In year",substr(d[j],1,4),
             "month",  substr(d[j],5,2),
             "string abc",
             "was seen",c[d[j]],"times"}' $InFile >$OutFile

... produced this OutFile ...

Code:

In year 2013 month 09 string abc was seen 2 times
In year 2013 month 10 string abc was seen 2 times
In year 2013 month 11 string abc was seen 4 times

Daniel B. Martin

byran cheung · 11-01-2013, 02:18 AM

Quote:

Originally Posted by Firerat

adapt a little

Code:

Total=0
MyString="abc"
for File in *.txt;do
    Count=$(awk 'BEGIN{C=0}{for (i=1;i<=NF;i++) if ( $i ~ '${MyString}' ){C++}} END{print C}' "$File")
    echo "No. of \"${MyString}\" in $(date "+%b %Y" --date="${File:4:6}01") = $Count"
    Total=$(( $Count + $Total ))
done
echo "Total No. of \"${MyString}\" = $Total"

Hi ,

I tested the script , it seems count the no. of lines , but not count the occurence of the "abc" , would you please check ? very thanks

Firerat · 11-01-2013, 06:02 AM

oops, I should have tested it

the prblem was

if ( $i ~ '${MyString}' )

should be

if ( $i ~ /'${MyString}'/ )

Code:

Total=0
MyString="abc"
for File in *.txt;do
    Count=$(awk 'BEGIN{C=0}{for (i=1;i<=NF;i++) if ( $i ~ /'${MyString}'/ ){C++}} END{print C}' "$File")
    echo "No. of \"${MyString}\" in $(date "+%b %Y" --date="${File:4:6}01") = $Count"
    Total=$(( $Count + $Total ))
done
echo "Total No. of \"${MyString}\" = $Total"

I used this to test

Code:

In my system , there are text files will be generated monthly , the file name begins with xxx , then year , month ( for example xxxxx201310.txt means Oct 2013 )

I have below command to count how many abc in the month , but it only count the number in this month .

NUMBER=$(cat xxxxx201310.txt |grep -c -s "abc" )

Could advise if I would like to have the following report , it shows the no. of abc for each month , and the total no. of abc , how to make it ? very thanks

no of abc in Oct 2013 = 111 no of abc in Nov 2013 = 222 no of abc in Dec 2013 = 333
"
"
"
Total no of abc = 999

the awk gives you 8, whereas grep -c results in 5

Firerat · 11-01-2013, 06:41 AM

actually, a much simpler count would be

Code:

Count=$( grep -o "${MyString}"  "$File" | wc -w )

the grep -o, only prints the pattern matches, the wc -w, is counting 'words'

actually, wc -w or wc -l ( lines ) will do

output from test file

Code:

grep -o abc xxxx201301.txt 
abc
abc
abc
abc
abc
abc
abc
abc

it will even split up, "abcabc" , which the awk fails to do