LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - General (https://www.linuxquestions.org/questions/linux-general-1/)
-   -   grep -w E (https://www.linuxquestions.org/questions/linux-general-1/grep-w-e-797568/)

dnaqvi 03-24-2010 11:17 AM
grep -w E
 
Hello All,

We are getting many messages in the log file.
There are many types of messages,
i.e. W for warning, E for error etc,

We wanted to grep errors contain only error type "E".
If I grep E it is greping all words which has letter E.

I trid to do below command but its getting repeating values.

grep -w E SystemOut.log | sort | uniq >> sysfinal2

Please help me for the write command.

thanks

slinx 03-24-2010 12:17 PM
You can replace
Code:
sort | uniq
with
Code:
sort -u
It would help if you posted examples of what you are getting with your command. Most likely you are seeing multiple lines for the same error, because the error text, or the timestamps, differ.

dnaqvi 03-24-2010 02:09 PM
grep -w E SystemOut.log | sort -u >> sysfinal2


I am getting repaeting values (only time stamps is different)

Here is one of the message:

[2/6/10 9:38:24:485 PST] 00000012 WSRdbXaResour E DSRA0302E: XAException occurred. Error code is: XA_RBOTHER (104). Exception is: <null>

[2/6/10 9:39:04:505 PST] 00000012 WSRdbXaResour E DSRA0302E: XAException occurred. Error code is: XA_RBOTHER (104). Exception is: <null>

thanks

blacky_5251 03-25-2010 05:10 PM
Try grepping for " E " instead of just "E" - i.e. space E space in quotes.

dnaqvi 03-26-2010 09:34 AM
Quote:
Originally Posted by blacky_5251 (Post 3912297)
Try grepping for " E " instead of just "E" - i.e. space E space in quotes.
------------------------------------------

grep -w " E " SystemOut.log sort -u >> sysfinal3



grep: sort: No such file or directory

------------------------------

grep -w " E " SystemOut.log >> sysfinal3



size of the file is zero.

dnaqvi 03-26-2010 09:45 AM
grep -w " A " SystemOut.log | sort | uniq >> sysfinal3


>cat sysfinal3

[2/6/10 9:39:15:007 PST] 0000aa21 AdminHelper A ADMN1020I: An attempt is made to stop the WebSphere_Portal server. (User ID = defaultWIMFileBasedRealm/admin)

This is the very first “ A “ type message in the SystemOut.log

But there are many more A types.
here are some of:

[2/6/10 9:39:20:439 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: Personalization_Workspace_6

[2/6/10 9:39:23:857 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: Personalization_Workspace_6

[2/6/10 9:39:23:859 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_TplApp

[2/6/10 9:39:24:970 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_TplApp

[2/6/10 9:39:24:971 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_DynamicUIApp

[2/6/10 9:39:25:420 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_DynamicUIApp

[2/6/10 9:39:25:421 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_WCM_Admin

[2/6/10 9:39:26:520 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_WCM_Admin

[2/6/10 9:39:26:521 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_WCMLocalRendering

[2/6/10 9:39:27:869 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_WCMLocalRendering

[2/6/10 9:39:27:870 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_WCM_Authoring_UI

[2/6/10 9:39:29:225 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_WCM_Authoring_UI

[2/6/10 9:39:29:226 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_Resource_Manager

[2/6/10 9:39:30:410 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_Resource_Manager

[2/6/10 9:39:30:412 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_Login_Portlet_App

Thanks

blacky_5251 03-26-2010 02:51 PM
Sorry I gave you a silly answer before. The -w option in grep does what I was trying to do with " E ", and I didn't properly grasp your problem.

Try this command:-
Code:
grep -w E SystemOut.log | sort | uniq -f 3
The "-f 3" option to uniq tells it to ignore the first three fields when determining the uniqueness of each record.

dnaqvi 03-26-2010 10:49 PM
Quote:
Originally Posted by blacky_5251 (Post 3913646)
Sorry I gave you a silly answer before. The -w option in grep does what I was trying to do with " E ", and I didn't properly grasp your problem.

Try this command:-
Code:
grep -w E SystemOut.log | sort | uniq -f 3
The "-f 3" option to uniq tells it to ignore the first three fields when determining the uniqueness of each record.
-----------------------------------------------------------------

If I need to track multiple types.

On above command we are considering only E types but may be want W types too.
What to do?

Thanks

dnaqvi 03-26-2010 11:25 PM
Thanks.

This works

grep -w E SystemOut.log | sort | uniq -f 3

1) If I like to add more message types as below:


grep -w E A W SystemOut.log | sort | uniq -f 3

will it work?

2) Can I add total quantity of error on the same output?


LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap: Total Appearance = 10

In SystemOut.log this message appeared 10 times.

D

rigor 03-27-2010 12:03 AM
Quote:
Originally Posted by dnaqvi (Post 3914015)
[...]
grep -w E A W SystemOut.log | sort | uniq -f 3
The following example would give any lines containing either W or E or A where there is a space before and after the character you are trying to match.

Quote:
egrep '( W )|( E )|( A )' SystemOut.log | sort | uniq -f 3

blacky_5251 03-27-2010 01:25 AM
Code:
grep -w [EAW]
Will locate all E, A and W errors.

blacky_5251 03-27-2010 01:32 AM
To count the number of error and warnings (in total), you could do this:-
Code:
CountErrors=$(grep -w E SystemOut.log | wc -l)
CountWarnings=$(grep -w W SystemOut.log | wc -l)
echo The number of errors is $CountErrors and the number of warnings is $CountWarnings
To count the number of different error types, try adding the -c option to the uniq command - i.e. uniq -c -f 3.

dnaqvi 03-27-2010 03:59 PM
Quote:
Originally Posted by blacky_5251 (Post 3914103)
To count the number of error and warnings (in total), you could do this:-
Code:
CountErrors=$(grep -w E SystemOut.log | wc -l)
CountWarnings=$(grep -w W SystemOut.log | wc -l)
echo The number of errors is $CountErrors and the number of warnings is $CountWarnings
To count the number of different error types, try adding the -c option to the uniq command - i.e. uniq -c -f 3.
-----

thanks
i will check monday morning usa (pst) time.

Many Thanks

dnaqvi 03-29-2010 09:15 AM
Uniq string is not working for for the multiple values

> grep -w [AI] SystemOut.log | sort | uniq -f 3

[3/29/10 0:01:45:464 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.
[3/29/10 5:00:10:117 PDT] 000000a0 CacheServiceI I DYNA1001I: WebSphere Dynamic Cache instance named services/cache/iwk/abspath initialized successfully.
[3/29/10 5:00:10:180 PDT] 000000a0 CacheServiceI I DYNA1001I: WebSphere Dynamic Cache instance named services/cache/iwk/abspathreverse initialized successfully.
[3/29/10 5:00:35:120 PDT] 0000009f CacheServiceI I DYNA1001I: WebSphere Dynamic Cache instance named services/cache/iwk/menu initialized successfully.
[3/29/10 5:01:46:113 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.
dn@dev2:/opt/IBM/WebSphere/PortalServer/log>

dnaqvi 03-29-2010 10:33 AM
I have noticed that single string is not working too.

> grep -w E SystemOut.log | sort | uniq -f 3



[3/29/10 5:37:32:540 PDT] 000000fe ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/index.jsp. Exception thrown : java.lang.NullPointerException
[3/29/10 7:03:14:566 PDT] 00000101 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/index.jsp. Exception thrown : java.lang.NullPointerException
[3/29/10 7:13:30:308 PDT] 00000120 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/index.jsp. Exception thrown : java.lang.NullPointerException
[3/29/10 5:37:32:599 PDT] 000000fe ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException
[3/29/10 8:02:01:636 PDT] 00000124 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException:
[3/29/10 7:13:30:335 PDT] 00000120 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException
[3/29/10 7:03:14:586 PDT] 00000101 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException
[3/29/10 8:02:01:618 PDT] 00000124 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/007recentInformation.jsp. Exception thrown : javax.servlet.ServletException: A required


[3/29/10 5:37:32:652 PDT] 000000fe ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: java.lang.NullPointerException
[3/29/10 7:13:30:358 PDT] 00000120 ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: java.lang.NullPointerException
[3/29/10 7:03:14:606 PDT] 00000101 ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: java.lang.NullPointerException
[3/29/10 8:02:01:629 PDT] 00000124 ServletWrappe E SRVE0014E: Uncaught service() exception root cause /jsp/007recentInformation.jsp: com.autonomy.aci.exceptions.MissingParameterException: A required
[3/29/10 8:02:01:643 PDT] 00000124 ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: javax.portlet.PortletException:

[3/29/10 2:19:24:510 PDT] 00000104 PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.
[3/29/10 5:37:32:700 PDT] 000000fe PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.
[3/29/10 7:03:14:624 PDT] 00000101 PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.
[3/29/10 7:13:30:372 PDT] 00000120 PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.



[3/29/10 3:03:16:239 PDT] 00000103 RepositorySer E Requested object could not be found.
[3/29/10 3:03:16:298 PDT] 00000101 RepositorySer E Requested object could not be found.
[3/29/10 5:28:25:269 PDT] 00000122 RepositorySer E Requested object could not be found.
[3/29/10 5:37:32:596 PDT] 000000fe LocalTranCoor E WLTC0017E: Resources rolled back due to setRollbackOnly() being called.
[3/29/10 7:03:14:584 PDT] 00000101 LocalTranCoor E WLTC0017E: Resources rolled back due to setRollbackOnly() being called.
[3/29/10 7:13:30:333 PDT] 00000120 LocalTranCoor E WLTC0017E: Resources rolled back due to setRollbackOnly() being called.

dnaqvi 03-29-2010 10:57 AM
I can get total number of error by this command
I have run this command twice.

(date; egrep "E" /opt/IBM/WebSphere/PortalServer/log/SystemOut.log|wc -l) >> /home/dn/total1.log 2>&1

more total1.log
Mon Mar 29 08:53:07 PDT 2010
553
Mon Mar 29 08:54:08 PDT 2010
607

I need ur assistance on previous issue as below.

grep -w E SystemOut.log | sort | uniq -f 3

blacky_5251 03-29-2010 02:39 PM
The problem is the sort I think. You need to read "man sort" and identify the correct syntax to use so you sort the text from the error, ignoring the timestamp data at the start of the line. "sort --key=3" might be what you need, but I got better results using 6 in the "sort --key=" and "uniq -f" values. Play around with the values to see what gives you the answer you're looking for :)

rigor 03-29-2010 03:15 PM
If I understand what you want, then looking at some of your log lines in more detail, anything like grep used to look for a capital A as a phrase by itself, without restricting the capital A to a specific field in the line, will find a line like this:

[3/29/10 8:02:01:618 PDT] 00000124 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/007recentInformation.jsp. Exception thrown : javax.servlet.ServletException: A required

because of the capital A as a separate phrase second from the end of the line.

Also, the log file lines don't have an entirely consistent format. It almost looks like most of them have a format something like:

Code:
[date time time-zone]  relative-address  process-name  error-class  error-code:  description:  detail
I'm not sure if the one field is a relative-address, or what all the fields truly are, I'm just guessing.

But not all of the log lines conform to a format very closely like that, some are quite different.

Sadly, we're probably going have to be very specific in the pattern, which is to say more complicated.

I've collected the example log file lines you're provided into a file named sys.log. This command seems to work to isolate the correct lines for the A type messages, because it requires that what I'm calling the error-class, A, W, E etc., is the sixth field in the line:

Code:
egrep '^((([^[:space:]]+)([[:space:]]+)){5}(A))' sys.log
it produces this output:

[3/29/10 0:01:45:464 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.
[3/29/10 5:01:46:113 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.
[2/6/10 9:39:20:439 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: Personalization_Workspace_6
[2/6/10 9:39:23:857 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: Personalization_Workspace_6
[2/6/10 9:39:23:859 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_TplApp
[2/6/10 9:39:24:970 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_TplApp
[2/6/10 9:39:24:971 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_DynamicUIApp
[2/6/10 9:39:25:420 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_DynamicUIApp
[2/6/10 9:39:25:421 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_WCM_Admin
[2/6/10 9:39:26:520 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_WCM_Admin
[2/6/10 9:39:26:521 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_WCMLocalRendering
[2/6/10 9:39:27:869 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_WCMLocalRendering
[2/6/10 9:39:27:870 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_WCM_Authoring_UI
[2/6/10 9:39:29:225 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_WCM_Authoring_UI
[2/6/10 9:39:29:226 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_Resource_Manager
[2/6/10 9:39:30:410 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_Resource_Manager
[2/6/10 9:39:30:412 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_Login_Portlet_App
[3/29/10 0:01:45:464 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.
[3/29/10 5:01:46:113 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.


Replacing the capital A in the command with A|E meaning A or E so that the command looks like this:

Code:
egrep '^((([^[:space:]]+)([[:space:]]+)){5}(A|E))' sys.log
it produces this output:

[3/29/10 5:37:32:540 PDT] 000000fe ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/index.jsp. Exception thrown : java.lang.NullPointerException
[3/29/10 7:03:14:566 PDT] 00000101 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/index.jsp. Exception thrown : java.lang.NullPointerException
[3/29/10 7:13:30:308 PDT] 00000120 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/index.jsp. Exception thrown : java.lang.NullPointerException
[3/29/10 5:37:32:599 PDT] 000000fe ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException
[3/29/10 8:02:01:636 PDT] 00000124 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException:
[3/29/10 7:13:30:335 PDT] 00000120 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException
[3/29/10 7:03:14:586 PDT] 00000101 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException
[3/29/10 8:02:01:618 PDT] 00000124 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/007recentInformation.jsp. Exception thrown : javax.servlet.ServletException: A required
[3/29/10 5:37:32:652 PDT] 000000fe ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: java.lang.NullPointerException
[3/29/10 7:13:30:358 PDT] 00000120 ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: java.lang.NullPointerException
[3/29/10 7:03:14:606 PDT] 00000101 ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: java.lang.NullPointerException
[3/29/10 8:02:01:629 PDT] 00000124 ServletWrappe E SRVE0014E: Uncaught service() exception root cause /jsp/007recentInformation.jsp: com.autonomy.aci.exceptions.MissingParameterException: A required
[3/29/10 8:02:01:643 PDT] 00000124 ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: javax.portlet.PortletException:
[3/29/10 2:19:24:510 PDT] 00000104 PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.
[3/29/10 5:37:32:700 PDT] 000000fe PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.
[3/29/10 7:03:14:624 PDT] 00000101 PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.
[3/29/10 7:13:30:372 PDT] 00000120 PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.
[3/29/10 3:03:16:239 PDT] 00000103 RepositorySer E Requested object could not be found.
[3/29/10 3:03:16:298 PDT] 00000101 RepositorySer E Requested object could not be found.
[3/29/10 5:28:25:269 PDT] 00000122 RepositorySer E Requested object could not be found.
[3/29/10 5:37:32:596 PDT] 000000fe LocalTranCoor E WLTC0017E: Resources rolled back due to setRollbackOnly() being called.
[3/29/10 7:03:14:584 PDT] 00000101 LocalTranCoor E WLTC0017E: Resources rolled back due to setRollbackOnly() being called.
[3/29/10 7:13:30:333 PDT] 00000120 LocalTranCoor E WLTC0017E: Resources rolled back due to setRollbackOnly() being called.
[3/29/10 0:01:45:464 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.
[3/29/10 5:01:46:113 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.
[2/6/10 9:39:20:439 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: Personalization_Workspace_6
[2/6/10 9:39:23:857 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: Personalization_Workspace_6
[2/6/10 9:39:23:859 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_TplApp
[2/6/10 9:39:24:970 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_TplApp
[2/6/10 9:39:24:971 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_DynamicUIApp
[2/6/10 9:39:25:420 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_DynamicUIApp
[2/6/10 9:39:25:421 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_WCM_Admin
[2/6/10 9:39:26:520 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_WCM_Admin
[2/6/10 9:39:26:521 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_WCMLocalRendering
[2/6/10 9:39:27:869 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_WCMLocalRendering
[2/6/10 9:39:27:870 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_WCM_Authoring_UI
[2/6/10 9:39:29:225 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_WCM_Authoring_UI
[2/6/10 9:39:29:226 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_Resource_Manager
[2/6/10 9:39:30:410 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_Resource_Manager
[2/6/10 9:39:30:412 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_Login_Portlet_App
[3/29/10 0:01:45:464 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.
[[3/29/10 5:37:32:540 PDT] 000000fe ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/index.jsp. Exception thrown : java.lang.NullPointerException
[3/29/10 7:03:14:566 PDT] 00000101 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/index.jsp. Exception thrown : java.lang.NullPointerException
[3/29/10 7:13:30:308 PDT] 00000120 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/index.jsp. Exception thrown : java.lang.NullPointerException
[3/29/10 5:37:32:599 PDT] 000000fe ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException
[3/29/10 8:02:01:636 PDT] 00000124 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException:
[3/29/10 7:13:30:335 PDT] 00000120 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException
[3/29/10 7:03:14:586 PDT] 00000101 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException
[3/29/10 8:02:01:618 PDT] 00000124 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/007recentInformation.jsp. Exception thrown : javax.servlet.ServletException: A required
[3/29/10 5:37:32:652 PDT] 000000fe ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: java.lang.NullPointerException
[3/29/10 7:13:30:358 PDT] 00000120 ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: java.lang.NullPointerException
[3/29/10 7:03:14:606 PDT] 00000101 ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: java.lang.NullPointerException
[3/29/10 8:02:01:629 PDT] 00000124 ServletWrappe E SRVE0014E: Uncaught service() exception root cause /jsp/007recentInformation.jsp: com.autonomy.aci.exceptions.MissingParameterException: A required
[3/29/10 8:02:01:643 PDT] 00000124 ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: javax.portlet.PortletException:
[3/29/10 2:19:24:510 PDT] 00000104 PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.
[3/29/10 5:37:32:700 PDT] 000000fe PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.
[3/29/10 7:03:14:624 PDT] 00000101 PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.
[3/29/10 7:13:30:372 PDT] 00000120 PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.
[3/29/10 3:03:16:239 PDT] 00000103 RepositorySer E Requested object could not be found.
[3/29/10 3:03:16:298 PDT] 00000101 RepositorySer E Requested object could not be found.
[3/29/10 5:28:25:269 PDT] 00000122 RepositorySer E Requested object could not be found.
[3/29/10 5:37:32:596 PDT] 000000fe LocalTranCoor E WLTC0017E: Resources rolled back due to setRollbackOnly() being called.
[3/29/10 7:03:14:584 PDT] 00000101 LocalTranCoor E WLTC0017E: Resources rolled back due to setRollbackOnly() being called.
[3/29/10 7:13:30:333 PDT] 00000120 LocalTranCoor E WLTC0017E: Resources rolled back due to setRollbackOnly() being called.
[3/29/10 5:01:46:113 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.

Even though the type of contents of the seventh and later fields isn't completely consistent, the file should still sort reasonably well, if we sort by the sixth field through the end of the line, then tell the sort command to select unique lines based on that. So adding the sort command with options to do that, we get this command sequence:

Code:
egrep '^((([^[:space:]]+)([[:space:]]+)){5}(A|E))' sys.log | sort -k6 -u
which produces this output:

[3/29/10 0:01:45:464 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.
[2/6/10 9:39:24:971 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_DynamicUIApp
[2/6/10 9:39:30:412 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_Login_Portlet_App
[2/6/10 9:39:29:226 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_Resource_Manager
[2/6/10 9:39:23:859 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_TplApp
[2/6/10 9:39:25:421 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_WCM_Admin
[2/6/10 9:39:27:870 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_WCM_Authoring_UI
[2/6/10 9:39:26:521 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_WCMLocalRendering
[2/6/10 9:39:20:439 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: Personalization_Workspace_6
[2/6/10 9:39:25:420 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_DynamicUIApp
[2/6/10 9:39:30:410 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_Resource_Manager
[2/6/10 9:39:24:970 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_TplApp
[2/6/10 9:39:26:520 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_WCM_Admin
[2/6/10 9:39:29:225 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_WCM_Authoring_UI
[2/6/10 9:39:27:869 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_WCMLocalRendering
[2/6/10 9:39:23:857 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: Personalization_Workspace_6
[3/29/10 2:19:24:510 PDT] 00000104 PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.
[3/29/10 3:03:16:239 PDT] 00000103 RepositorySer E Requested object could not be found.
[3/29/10 5:37:32:652 PDT] 000000fe ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: java.lang.NullPointerException
[3/29/10 8:02:01:643 PDT] 00000124 ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: javax.portlet.PortletException:
[3/29/10 8:02:01:629 PDT] 00000124 ServletWrappe E SRVE0014E: Uncaught service() exception root cause /jsp/007recentInformation.jsp: com.autonomy.aci.exceptions.MissingParameterException: A required
[3/29/10 5:37:32:599 PDT] 000000fe ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException
[3/29/10 8:02:01:636 PDT] 00000124 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException:
[3/29/10 8:02:01:618 PDT] 00000124 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/007recentInformation.jsp. Exception thrown : javax.servlet.ServletException: A required
[3/29/10 5:37:32:540 PDT] 000000fe ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/index.jsp. Exception thrown : java.lang.NullPointerException
[3/29/10 5:37:32:596 PDT] 000000fe LocalTranCoor E WLTC0017E: Resources rolled back due to setRollbackOnly() being called.

At a quick glance this might not look entirely correct, the lines from "3/29/10 5:37:32:599 PDT" and "3/29/10 8:02:01:636 PDT" look almost identical. But the second line as posted by you has a colon ":" at the end of the line, whereas the other doesn't.

Is that what you wanted, or did you want to ignore other fields as well, such as the fourth field?

Hope this helps.

dnaqvi 03-29-2010 04:13 PM
Quote:
Originally Posted by kakaka (Post 3917128)
If I understand what you want, then looking at some of your log lines in more detail, anything like grep used to look for a capital A as a phrase by itself, without restricting the capital A to a specific field in the line, will find a line like this:

[3/29/10 8:02:01:618 PDT] 00000124 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/007recentInformation.jsp. Exception thrown : javax.servlet.ServletException: A required

because of the capital A as a separate phrase second from the end of the line.

Also, the log file lines don't have an entirely consistent format. It almost looks like most of them have a format something like:

Code:
[date time time-zone]  relative-address  process-name  error-class  error-code:  description:  detail
I'm not sure if the one field is a relative-address, or what all the fields truly are, I'm just guessing.

But not all of the log lines conform to a format very closely like that, some are quite different.

Sadly, we're probably going have to be very specific in the pattern, which is to say more complicated.

I've collected the example log file lines you're provided into a file named sys.log. This command seems to work to isolate the correct lines for the A type messages, because it requires that what I'm calling the error-class, A, W, E etc., is the sixth field in the line:

Code:
egrep '^((([^[:space:]]+)([[:space:]]+)){5}(A))' sys.log
it produces this output:

[3/29/10 0:01:45:464 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.
[3/29/10 5:01:46:113 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.
[2/6/10 9:39:20:439 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: Personalization_Workspace_6
[2/6/10 9:39:23:857 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: Personalization_Workspace_6
[2/6/10 9:39:23:859 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_TplApp
[2/6/10 9:39:24:970 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_TplApp
[2/6/10 9:39:24:971 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_DynamicUIApp
[2/6/10 9:39:25:420 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_DynamicUIApp
[2/6/10 9:39:25:421 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_WCM_Admin
[2/6/10 9:39:26:520 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_WCM_Admin
[2/6/10 9:39:26:521 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_WCMLocalRendering
[2/6/10 9:39:27:869 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_WCMLocalRendering
[2/6/10 9:39:27:870 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_WCM_Authoring_UI
[2/6/10 9:39:29:225 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_WCM_Authoring_UI
[2/6/10 9:39:29:226 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_Resource_Manager
[2/6/10 9:39:30:410 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_Resource_Manager
[2/6/10 9:39:30:412 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_Login_Portlet_App
[3/29/10 0:01:45:464 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.
[3/29/10 5:01:46:113 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.


Replacing the capital A in the command with A|E meaning A or E so that the command looks like this:

Code:
egrep '^((([^[:space:]]+)([[:space:]]+)){5}(A|E))' sys.log
it produces this output:

[3/29/10 5:37:32:540 PDT] 000000fe ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/index.jsp. Exception thrown : java.lang.NullPointerException
[3/29/10 7:03:14:566 PDT] 00000101 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/index.jsp. Exception thrown : java.lang.NullPointerException
[3/29/10 7:13:30:308 PDT] 00000120 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/index.jsp. Exception thrown : java.lang.NullPointerException
[3/29/10 5:37:32:599 PDT] 000000fe ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException
[3/29/10 8:02:01:636 PDT] 00000124 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException:
[3/29/10 7:13:30:335 PDT] 00000120 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException
[3/29/10 7:03:14:586 PDT] 00000101 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException
[3/29/10 8:02:01:618 PDT] 00000124 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/007recentInformation.jsp. Exception thrown : javax.servlet.ServletException: A required
[3/29/10 5:37:32:652 PDT] 000000fe ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: java.lang.NullPointerException
[3/29/10 7:13:30:358 PDT] 00000120 ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: java.lang.NullPointerException
[3/29/10 7:03:14:606 PDT] 00000101 ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: java.lang.NullPointerException
[3/29/10 8:02:01:629 PDT] 00000124 ServletWrappe E SRVE0014E: Uncaught service() exception root cause /jsp/007recentInformation.jsp: com.autonomy.aci.exceptions.MissingParameterException: A required
[3/29/10 8:02:01:643 PDT] 00000124 ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: javax.portlet.PortletException:
[3/29/10 2:19:24:510 PDT] 00000104 PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.
[3/29/10 5:37:32:700 PDT] 000000fe PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.
[3/29/10 7:03:14:624 PDT] 00000101 PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.
[3/29/10 7:13:30:372 PDT] 00000120 PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.
[3/29/10 3:03:16:239 PDT] 00000103 RepositorySer E Requested object could not be found.
[3/29/10 3:03:16:298 PDT] 00000101 RepositorySer E Requested object could not be found.
[3/29/10 5:28:25:269 PDT] 00000122 RepositorySer E Requested object could not be found.
[3/29/10 5:37:32:596 PDT] 000000fe LocalTranCoor E WLTC0017E: Resources rolled back due to setRollbackOnly() being called.
[3/29/10 7:03:14:584 PDT] 00000101 LocalTranCoor E WLTC0017E: Resources rolled back due to setRollbackOnly() being called.
[3/29/10 7:13:30:333 PDT] 00000120 LocalTranCoor E WLTC0017E: Resources rolled back due to setRollbackOnly() being called.
[3/29/10 0:01:45:464 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.
[3/29/10 5:01:46:113 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.
[2/6/10 9:39:20:439 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: Personalization_Workspace_6
[2/6/10 9:39:23:857 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: Personalization_Workspace_6
[2/6/10 9:39:23:859 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_TplApp
[2/6/10 9:39:24:970 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_TplApp
[2/6/10 9:39:24:971 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_DynamicUIApp
[2/6/10 9:39:25:420 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_DynamicUIApp
[2/6/10 9:39:25:421 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_WCM_Admin
[2/6/10 9:39:26:520 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_WCM_Admin
[2/6/10 9:39:26:521 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_WCMLocalRendering
[2/6/10 9:39:27:869 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_WCMLocalRendering
[2/6/10 9:39:27:870 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_WCM_Authoring_UI
[2/6/10 9:39:29:225 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_WCM_Authoring_UI
[2/6/10 9:39:29:226 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_Resource_Manager
[2/6/10 9:39:30:410 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_Resource_Manager
[2/6/10 9:39:30:412 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_Login_Portlet_App
[3/29/10 0:01:45:464 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.
[[3/29/10 5:37:32:540 PDT] 000000fe ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/index.jsp. Exception thrown : java.lang.NullPointerException
[3/29/10 7:03:14:566 PDT] 00000101 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/index.jsp. Exception thrown : java.lang.NullPointerException
[3/29/10 7:13:30:308 PDT] 00000120 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/index.jsp. Exception thrown : java.lang.NullPointerException
[3/29/10 5:37:32:599 PDT] 000000fe ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException
[3/29/10 8:02:01:636 PDT] 00000124 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException:
[3/29/10 7:13:30:335 PDT] 00000120 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException
[3/29/10 7:03:14:586 PDT] 00000101 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException
[3/29/10 8:02:01:618 PDT] 00000124 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/007recentInformation.jsp. Exception thrown : javax.servlet.ServletException: A required
[3/29/10 5:37:32:652 PDT] 000000fe ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: java.lang.NullPointerException
[3/29/10 7:13:30:358 PDT] 00000120 ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: java.lang.NullPointerException
[3/29/10 7:03:14:606 PDT] 00000101 ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: java.lang.NullPointerException
[3/29/10 8:02:01:629 PDT] 00000124 ServletWrappe E SRVE0014E: Uncaught service() exception root cause /jsp/007recentInformation.jsp: com.autonomy.aci.exceptions.MissingParameterException: A required
[3/29/10 8:02:01:643 PDT] 00000124 ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: javax.portlet.PortletException:
[3/29/10 2:19:24:510 PDT] 00000104 PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.
[3/29/10 5:37:32:700 PDT] 000000fe PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.
[3/29/10 7:03:14:624 PDT] 00000101 PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.
[3/29/10 7:13:30:372 PDT] 00000120 PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.
[3/29/10 3:03:16:239 PDT] 00000103 RepositorySer E Requested object could not be found.
[3/29/10 3:03:16:298 PDT] 00000101 RepositorySer E Requested object could not be found.
[3/29/10 5:28:25:269 PDT] 00000122 RepositorySer E Requested object could not be found.
[3/29/10 5:37:32:596 PDT] 000000fe LocalTranCoor E WLTC0017E: Resources rolled back due to setRollbackOnly() being called.
[3/29/10 7:03:14:584 PDT] 00000101 LocalTranCoor E WLTC0017E: Resources rolled back due to setRollbackOnly() being called.
[3/29/10 7:13:30:333 PDT] 00000120 LocalTranCoor E WLTC0017E: Resources rolled back due to setRollbackOnly() being called.
[3/29/10 5:01:46:113 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.

Even though the type of contents of the seventh and later fields isn't completely consistent, the file should still sort reasonably well, if we sort by the sixth field through the end of the line, then tell the sort command to select unique lines based on that. So adding the sort command with options to do that, we get this command sequence:

Code:
egrep '^((([^[:space:]]+)([[:space:]]+)){5}(A|E))' sys.log | sort -k6 -u
which produces this output:

[3/29/10 0:01:45:464 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.
[2/6/10 9:39:24:971 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_DynamicUIApp
[2/6/10 9:39:30:412 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_Login_Portlet_App
[2/6/10 9:39:29:226 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_Resource_Manager
[2/6/10 9:39:23:859 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_TplApp
[2/6/10 9:39:25:421 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_WCM_Admin
[2/6/10 9:39:27:870 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_WCM_Authoring_UI
[2/6/10 9:39:26:521 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: PA_WCMLocalRendering
[2/6/10 9:39:20:439 PST] 00000012 ApplicationMg A WSVR0217I: Stopping application: Personalization_Workspace_6
[2/6/10 9:39:25:420 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_DynamicUIApp
[2/6/10 9:39:30:410 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_Resource_Manager
[2/6/10 9:39:24:970 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_TplApp
[2/6/10 9:39:26:520 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_WCM_Admin
[2/6/10 9:39:29:225 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_WCM_Authoring_UI
[2/6/10 9:39:27:869 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: PA_WCMLocalRendering
[2/6/10 9:39:23:857 PST] 00000012 ApplicationMg A WSVR0220I: Application stopped: Personalization_Workspace_6
[3/29/10 2:19:24:510 PDT] 00000104 PortletRender E com.ibm.wps.engine.tags.PortletRenderTag doStartTag EJPEJ0066E: The portlet could not be rendered.
[3/29/10 3:03:16:239 PDT] 00000103 RepositorySer E Requested object could not be found.
[3/29/10 5:37:32:652 PDT] 000000fe ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: java.lang.NullPointerException
[3/29/10 8:02:01:643 PDT] 00000124 ServletWrappe E SRVE0014E: Uncaught service() exception root cause AutonomyJSPPortlet: javax.portlet.PortletException:
[3/29/10 8:02:01:629 PDT] 00000124 ServletWrappe E SRVE0014E: Uncaught service() exception root cause /jsp/007recentInformation.jsp: com.autonomy.aci.exceptions.MissingParameterException: A required
[3/29/10 5:37:32:599 PDT] 000000fe ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException
[3/29/10 8:02:01:636 PDT] 00000124 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet AutonomyJSPPortlet. Exception thrown : javax.servlet.ServletException:
[3/29/10 8:02:01:618 PDT] 00000124 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/007recentInformation.jsp. Exception thrown : javax.servlet.ServletException: A required
[3/29/10 5:37:32:540 PDT] 000000fe ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet /jsp/index.jsp. Exception thrown : java.lang.NullPointerException
[3/29/10 5:37:32:596 PDT] 000000fe LocalTranCoor E WLTC0017E: Resources rolled back due to setRollbackOnly() being called.

At a quick glance this might not look entirely correct, the lines from "3/29/10 5:37:32:599 PDT" and "3/29/10 8:02:01:636 PDT" look almost identical. But the second line as posted by you has a colon ":" at the end of the line, whereas the other doesn't.

Is that what you wanted, or did you want to ignore other fields as well, such as the fourth field?

Hope this helps.
--------------------------

This looks better. thanks
I'd to use this logic in the script.
From the same script I want output of quantity of each type in previous script.

thanks

rigor 03-29-2010 06:32 PM
If you have perl available to you, it was simple to write a little program that adjusts itself to whatever types you choose to run through it, like this:


Code:
egrep '^((([^[:space:]]+)([[:space:]]+)){5}(A))' sys.log | sort -k6 -u | perl -w count_em.pl
Log message count for type 'A' is: 16.
Log message total count was: 16.

Code:
egrep '^((([^[:space:]]+)([[:space:]]+)){5}(A|E))' sys.log | sort -k6 -u | perl -w count_em.pl
Log message count for type 'A' is: 16.
Log message count for type 'E' is: 10.
Log message total count was: 26.

I wasn't able to upload the little perl program as an attachment, so I'll paste the code here. If you put this code in a file named count_em.pl, it will work with the command sequence shown above:

Code:

my  $total_count  =  0  ;
my  $input_line  =  ""  ;
my  @pieces  ;
my  %type_counts  ;


while(  $input_line = <STDIN>  )
{
    @pieces = split(  /[\s]+/ ,  $input_line  )  ;

    if (  ! exists  $type_counts{ $pieces[ 5 ] }  )
    {
        $type_counts{ $pieces[ 5 ] } = 0 ;
    }

    $type_counts{ $pieces[ 5 ] }  +=  1 ;

    ++$total_count ;
}


while (  (  $type ,  $count  )  =  (  each %type_counts  )  )
{
        print "Log message count for type \047$type\047 is: $count.\n"  ;
}


print "Log message total count was: $total_count.\n" ;
If you don't have perl, the program can be converted to awk. Or, you can just get each type individually, count them with wc -l as has already been mentioned by other folks, and add each type for a total if you need it:


Code:
error_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(E))' sys.log | sort -k6 -u | wc -l`

advisory_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(A))' sys.log | sort -k6 -u | wc -l`

total_count=`expr $advisory_count  \+  $error_count`

echo "There were $error_count error messages, $advisory_count advisory messages, $total_count together."
There were 10 error messages, 16 advisory messages, 26 together.


Although if you have any large number of messages to go through, it might be nice to be able to do all the counts in one pass. Otherwise it could be rather slow.

Hope this helps.

dnaqvi 03-29-2010 06:55 PM
Quote:
Originally Posted by kakaka (Post 3917273)
If you have perl available to you, it was simple to write a little program that adjusts itself to whatever types you choose to run through it, like this:


Code:
egrep '^((([^[:space:]]+)([[:space:]]+)){5}(A))' sys.log | sort -k6 -u | perl -w count_em.pl
Log message count for type 'A' is: 16.
Log message total count was: 16.

Code:
egrep '^((([^[:space:]]+)([[:space:]]+)){5}(A|E))' sys.log | sort -k6 -u | perl -w count_em.pl
Log message count for type 'A' is: 16.
Log message count for type 'E' is: 10.
Log message total count was: 26.

I wasn't able to upload the little perl program as an attachment, so I'll paste the code here. If you put this code in a file named count_em.pl, it will work with the command sequence shown above:

Code:

my  $total_count  =  0  ;
my  $input_line  =  ""  ;
my  @pieces  ;
my  %type_counts  ;


while(  $input_line = <STDIN>  )
{
    @pieces = split(  /[\s]+/ ,  $input_line  )  ;

    if (  ! exists  $type_counts{ $pieces[ 5 ] }  )
    {
        $type_counts{ $pieces[ 5 ] } = 0 ;
    }

    $type_counts{ $pieces[ 5 ] }  +=  1 ;

    ++$total_count ;
}


while (  (  $type ,  $count  )  =  (  each %type_counts  )  )
{
        print "Log message count for type \047$type\047 is: $count.\n"  ;
}


print "Log message total count was: $total_count.\n" ;
If you don't have perl, the program can be converted to awk. Or, you can just get each type individually, count them with wc -l as has already been mentioned by other folks, and add each type for a total if you need it:


Code:
error_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(E))' sys.log | sort -k6 -u | wc -l`

advisory_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(A))' sys.log | sort -k6 -u | wc -l`

total_count=`expr $advisory_count  \+  $error_count`

echo "There were $error_count error messages, $advisory_count advisory messages, $total_count together."
There were 10 error messages, 16 advisory messages, 26 together.


Although if you have any large number of messages to go through, it might be nice to be able to do all the counts in one pass. Otherwise it could be rather slow.

Hope this helps.
I do have perl.
I will try and update to you shortly. thanks

dnaqvi 03-30-2010 11:29 AM
Quote:
Originally Posted by dnaqvi (Post 3917285)
I do have perl.
I will try and update to you shortly. thanks
I did not use perl yet because I did not get it.

But I have used awk and get following results:

There were 10 error messages, 15 warning messages, 10 together.

Why 10 together?

I thought it should be 35, total of 10 plus 15.

I also need to join following command in the same script to get one report which will contain messages and quantity together.

egrep '^((([^[:space:]]+)([[:space:]]+)){5}(W|E))' SystemOut.log | sort -k6 -u

Thanks

dnaqvi 03-30-2010 11:52 AM
Quote:
Originally Posted by dnaqvi (Post 3918094)
I did not use perl yet because I did not get it.

But I have used awk and get following results:

There were 10 error messages, 15 warning messages, 10 together.

Why 10 together?

I thought it should be 35, total of 10 plus 15.

I also need to join following command in the same script to get one report which will contain messages and quantity together.

egrep '^((([^[:space:]]+)([[:space:]]+)){5}(W|E))' SystemOut.log | sort -k6 -u

Thanks
Now I am getting correct total.
It was a typo.

Open issue is as:
Same report with counts & messages.

thanks

dnaqvi 03-30-2010 02:39 PM
Now I am running following report (error_message.sh )

error_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(E))' path/log/SystemOut.log | sort -k6 -u | wc -l`

warning_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(W))' path/log/SystemOut.log | sort -k6 -u | wc -l`

update_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(W))' path/log/SystemOut.log | sort -k6 -u | wc -l`

action_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(W))' path/log/SystemOut.log | sort -k6 -u | wc -l`

total_count=`expr $warning_count \+ $error_count \+ $update_count \+ $action_count`

echo "There were $error_count error messages, $warning_count warning messages, $update_count update counts, $action_count action counts, $total_count together." >> message.log
--------------------------------------------------

REPORT:
cat message.log
There were 12 error messages, 20 warning messages, 20 update counts, 20 action counts, 72 together.

I have run this report 11.50 am.
Our cutoff log is incurred every 12 hours.
I have set cron job as below:

0 */1 * * * /home/dn/error_message.sh (runs everyy hour)
50 11,23 * * * /home/dn/MailTotal (send report to us via email)

Am I receive total error incurred in each hour or the total?

rigor 03-30-2010 11:27 PM
If what you've posted is accurate, it looks like you're counting warnings three times and assigning them to different variables.

You might want to adjust that. Also,

*/1 in a crontab is no different than *

I don't know your log cutoff works. Often if someone has a log file named SystemOut.log they might keep a copy of it by saving it with a new name like SystemOut.log.0 or SystemOut.log.1 or whatever. Sometimes that saves log files are compressed to save space. If yours works anything like that, it were me, I'd insert the code to count log message types into the procedure to do the log cutoff, so it would count the full period covered by the log, rather than ten minutes less than the full period.

As to which count you're getting, I was under the impression that you wanted to ignore the time stamps in the log messages. So the pattern does ignore the time stamps. If you have a log for 12 hours, whenever you run the commands, it will count all the messages in the log.

If you want them hourly, you could change the pattern used by the grep command to select those messages from a specific hour.

dnaqvi 03-31-2010 08:48 AM
Quote:
Originally Posted by kakaka (Post 3918669)
If what you've posted is accurate, it looks like you're counting warnings three times and assigning them to different variables.

You might want to adjust that. Also,

*/1 in a crontab is no different than *

I don't know your log cutoff works. Often if someone has a log file named SystemOut.log they might keep a copy of it by saving it with a new name like SystemOut.log.0 or SystemOut.log.1 or whatever. Sometimes that saves log files are compressed to save space. If yours works anything like that, it were me, I'd insert the code to count log message types into the procedure to do the log cutoff, so it would count the full period covered by the log, rather than ten minutes less than the full period.

As to which count you're getting, I was under the impression that you wanted to ignore the time stamps in the log messages. So the pattern does ignore the time stamps. If you have a log for 12 hours, whenever you run the commands, it will count all the messages in the log.

If you want them hourly, you could change the pattern used by the grep command to select those messages from a specific hour.
we do have our old SystemOut.log like SystemOut_10.03.30_12.00.00.
Our basic reason to do this program to track on errors and warnings. If you have any suggestion for me, please share with me.
I have just run this program in our dev environment. Right now in our SystemOut.lg file we have only "A" type. But I got output as below:
I run twice and get entries twice as below:

There were 0 error messages, 0 warning messages, 0 update counts, 0 action counts, 0 together.
There were 0 error messages, 0 warning messages, 0 update counts, 0 action counts, 0 together.

It not picking up type "A"

dnaqvi 03-31-2010 08:51 AM
Quote:
Originally Posted by dnaqvi (Post 3919167)
we do have our old SystemOut.log like SystemOut_10.03.30_12.00.00.
Our basic reason to do this program to track on errors and warnings. If you have any suggestion for me, please share with me.
I have just run this program in our dev environment. Right now in our SystemOut.lg file we have only "A" type. But I got output as below:
I run twice and get entries twice as below:

There were 0 error messages, 0 warning messages, 0 update counts, 0 action counts, 0 together.
There were 0 error messages, 0 warning messages, 0 update counts, 0 action counts, 0 together.

It not picking up type "A"
Here are the entries from SystemOut.log

[3/31/10 0:35:36:968 PDT] 00000097 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00.111
[3/31/10 1:35:37:098 PDT] 00000097 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00.111
[3/31/10 2:35:37:251 PDT] 00000097 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00.111
[3/31/10 3:35:37:387 PDT] 00000097 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00.111
[3/31/10 4:35:37:515 PDT] 00000097 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00.111
[3/31/10 5:35:37:641 PDT] 00000097 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00.111
[3/31/10 6:35:37:771 PDT] 00000097 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00.111

Here is the Program:

error_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(E))' /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/SystemOut.log | sort -k6 -u | wc -l`

warning_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(W))' /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/SystemOut.log | sort -k6 -u | wc -l`

update_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(W))' /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/SystemOut.log | sort -k6 -u | wc -l`

action_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(W))' /opt/IBM/WebSphere/wp_profile/logs/WebSphere_Portal/SystemOut.log | sort -k6 -u | wc -l`

total_count=`expr $warning_count \+ $error_count \+ $update_count \+ $action_count`

echo "There were $error_count error messages, $warning_count warning messages, $update_count update counts, $action_count action counts, $total_count together." >> message.log

rigor 03-31-2010 07:18 PM
The following four commands are what you posted as what you are running.

Code:
error_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(E))' path/log/SystemOut.log | sort -k6 -u | wc -l`

warning_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(W))' path/log/SystemOut.log | sort -k6 -u | wc -l`

update_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(W))' path/log/SystemOut.log | sort -k6 -u | wc -l`

action_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(W))' path/log/SystemOut.log | sort -k6 -u | wc -l`
The single capital letter placed inside parentheses determines what type of message you select. So the first command selects messages with a single captial E in the sixth field. The other three commands all select messages with a single capital "W" in the sixth field. Since you don't have an A instead of a W, you're not looking for messages with an A.

I'm not entirely sure at this point which capital letter you are using for which type of message. I'm guessing W means warning, A means action, U means update, and E means. If the letters you are using are different, then you need to use those letters instead. But if the letters I listed are the actual letters you are using, then you would need commands like this:

Code:
error_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(E))' path/log/SystemOut.log | sort -k6 -u | wc -l`
                                                        ^ E goes here for errors

warning_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(W))' path/log/SystemOut.log | sort -k6 -u | wc -l`
                                                          ^ W goes here for warnings


update_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(U))' path/log/SystemOut.log | sort -k6 -u | wc -l`
                                                          ^ U goes here for updates
 
action_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(A))' path/log/SystemOut.log | sort -k6 -u | wc -l`
                                                          ^ A goes here for action messages.
One way or another you need to put the single capital letter associated with the messages you want to count, into the command to count that type of message.

If those are not the letters you use, then you need to replace them with the letters you actually use.

dnaqvi 04-01-2010 09:01 AM
Quote:
Originally Posted by kakaka (Post 3919783)
The following four commands are what you posted as what you are running.

Code:
error_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(E))' path/log/SystemOut.log | sort -k6 -u | wc -l`

warning_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(W))' path/log/SystemOut.log | sort -k6 -u | wc -l`

update_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(W))' path/log/SystemOut.log | sort -k6 -u | wc -l`

action_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(W))' path/log/SystemOut.log | sort -k6 -u | wc -l`
The single capital letter placed inside parentheses determines what type of message you select. So the first command selects messages with a single captial E in the sixth field. The other three commands all select messages with a single capital "W" in the sixth field. Since you don't have an A instead of a W, you're not looking for messages with an A.

I'm not entirely sure at this point which capital letter you are using for which type of message. I'm guessing W means warning, A means action, U means update, and E means. If the letters you are using are different, then you need to use those letters instead. But if the letters I listed are the actual letters you are using, then you would need commands like this:

Code:
error_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(E))' path/log/SystemOut.log | sort -k6 -u | wc -l`
                                                        ^ E goes here for errors

warning_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(W))' path/log/SystemOut.log | sort -k6 -u | wc -l`
                                                          ^ W goes here for warnings


update_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(U))' path/log/SystemOut.log | sort -k6 -u | wc -l`
                                                          ^ U goes here for updates
 
action_count=`egrep '^((([^[:space:]]+)([[:space:]]+)){5}(A))' path/log/SystemOut.log | sort -k6 -u | wc -l`
                                                          ^ A goes here for action messages.
One way or another you need to put the single capital letter associated with the messages you want to count, into the command to count that type of message.

If those are not the letters you use, then you need to replace them with the letters you actually use.
yes E for error, W for warning, I for updates and A for actions.

Our systemout.log cuttoff time is 12 hr.
I want entries generate every hour in the report.
We set cron job to send us report 11.50 am/pm.
How can I get report generate very hour?
Is it via cron job, if yes what should be the entry?

thanks

dnaqvi 04-01-2010 09:06 AM
I it ok?


0 * * * * /home/dn/error_message.

will it generate every hour?

rigor 04-01-2010 03:21 PM
Quote:
Originally Posted by dnaqvi (Post 3920425)
I it ok?


0 * * * * /home/dn/error_message.

will it generate every hour?
Yes, if there's no period at the end of the crontab line, that will run the error_message program every hour.

But, it will count ALL the messages in the file. Not just those from a single hour. Is that what you want?

dnaqvi 04-05-2010 11:22 AM
Quote:
Originally Posted by kakaka (Post 3920822)
Yes, if there's no period at the end of the crontab line, that will run the error_message program every hour.

But, it will count ALL the messages in the file. Not just those from a single hour. Is that what you want?
Yes or no.
Thing has change on demand.

What about every hour?

But if restart server before cutoff time,
the error counts will start from zero but we want total errors every hour.

for eg:

1hr -- 20 counts
2hr -- 45 counts (25 from current hours and 20 from previos hr)
3hr -- 45 counts (0 from current hours and 45 from previous hrs)

Thanks

rigor 04-06-2010 12:14 AM
Hi dnaqvi,

If understand correctly you've stated that you start a new log file every 12 hours.

The pattern we've been talking about will count the overall total of whatever type of messages you specify, in a single log file, each time you use the pattern, without any concern for the hour in the message .

Let's say that you are able to use the pattern in a script, and manage to run the script, exactly at the time, after the last message for one hour is put in the log file, and before the first message for the next hour is put in the log file. Then, for a particular type of message, with this list of counts for the first few hours of the day as an example, the counts would work like this:

Code:
hour    count which occurred *that* hour    pattern would return this count
====    ==================================  ===============================
  0                                    10                              10
  1                                    15                              25
  2                                    25                              50
  3                                    10                              60

It will only do that for a single log file. As soon as you start a new log file, the count returned by a pattern used for a particular type of message, will start over at zero.

So, if you want a continuously increasing count, then you'll need to save the count you got from one log file, and add it to the count from the next log file.

If for some reason, it's easier for you to add up the total, by getting the count for each specific hour according to the time in the message for a particular type of message, that can be done using commands like these:

Code:
$ the_hour=`date +%H`
$ echo $the_hour
23
$ egrep '^(([^\ ]+)([\ ]+))'${the_hour}'(((:[0-9]+)){3})((([\ ]+)([^\ ]+)){3})([\ ]+)(A)' sys.log
[3/29/10 23:01:46:113 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.
Again, I'm using the log messages you've provided, but I changed the hour in some of them so that there are some messages for each hour of some day. When I started typing this it was between the 23rd hour ( 11 PM ) and the end of the day, in the local time zone. So the special format on the date command, just outputs the hour of the day from 0 through 23, in this case 23, which is then assigned to variable the_hour. If you put the command in the script, it could used to select the same hour for each pattern for each type of message.

If you were to use that with what we'd talked about before, and just using only error and advisory messages as an example, with the pattern for a specific hour according to the time in the message, it might look something like:

Code:
the_hour=`date +%H`

error_count=`egrep '^(([^\ ]+)([\ ]+))'${the_hour}'(((:[0-9]+)){3})((([\ ]+)([^\ ]+)){3})([\ ]+)(E)' sys.log | sort -k6 -u | wc -l`

advisory_count=`egrep '^(([^\ ]+)([\ ]+))'${the_hour}'(((:[0-9]+)){3})((([\ ]+)([^\ ]+)){3})([\ ]+)(A)' sys.log | sort -k6 -u | wc -l`

total_count=`expr $advisory_count  \+  $error_count`

echo "There were $error_count error messages, $advisory_count advisory messages, $total_count together."
But in case you did need to do something like that, please keep in mind that by using the date command, the patterns would look for messages from whatever hour it is when the script runs, even if it's run near the end of an hour, something delays it's running, and it actually runs just after the start of a new hour.

You could also pass in the hour for which you want to search to be absolutely sure you get the right hour.

Whichever way you need to do things, hope this helps.

dnaqvi 04-07-2010 08:47 AM
Quote:
Originally Posted by kakaka (Post 3925712)
Hi dnaqvi,

If understand correctly you've stated that you start a new log file every 12 hours.

The pattern we've been talking about will count the overall total of whatever type of messages you specify, in a single log file, each time you use the pattern, without any concern for the hour in the message .

Let's say that you are able to use the pattern in a script, and manage to run the script, exactly at the time, after the last message for one hour is put in the log file, and before the first message for the next hour is put in the log file. Then, for a particular type of message, with this list of counts for the first few hours of the day as an example, the counts would work like this:

Code:
hour    count which occurred *that* hour    pattern would return this count
====    ==================================  ===============================
  0                                    10                              10
  1                                    15                              25
  2                                    25                              50
  3                                    10                              60

It will only do that for a single log file. As soon as you start a new log file, the count returned by a pattern used for a particular type of message, will start over at zero.

So, if you want a continuously increasing count, then you'll need to save the count you got from one log file, and add it to the count from the next log file.

If for some reason, it's easier for you to add up the total, by getting the count for each specific hour according to the time in the message for a particular type of message, that can be done using commands like these:

Code:
$ the_hour=`date +%H`
$ echo $the_hour
23
$ egrep '^(([^\ ]+)([\ ]+))'${the_hour}'(((:[0-9]+)){3})((([\ ]+)([^\ ]+)){3})([\ ]+)(A)' sys.log
[3/29/10 23:01:46:113 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.
Again, I'm using the log messages you've provided, but I changed the hour in some of them so that there are some messages for each hour of some day. When I started typing this it was between the 23rd hour ( 11 PM ) and the end of the day, in the local time zone. So the special format on the date command, just outputs the hour of the day from 0 through 23, in this case 23, which is then assigned to variable the_hour. If you put the command in the script, it could used to select the same hour for each pattern for each type of message.

If you were to use that with what we'd talked about before, and just using only error and advisory messages as an example, with the pattern for a specific hour according to the time in the message, it might look something like:

Code:
the_hour=`date +%H`

error_count=`egrep '^(([^\ ]+)([\ ]+))'${the_hour}'(((:[0-9]+)){3})((([\ ]+)([^\ ]+)){3})([\ ]+)(E)' sys.log | sort -k6 -u | wc -l`

advisory_count=`egrep '^(([^\ ]+)([\ ]+))'${the_hour}'(((:[0-9]+)){3})((([\ ]+)([^\ ]+)){3})([\ ]+)(A)' sys.log | sort -k6 -u | wc -l`

total_count=`expr $advisory_count  \+  $error_count`

echo "There were $error_count error messages, $advisory_count advisory messages, $total_count together."
But in case you did need to do something like that, please keep in mind that by using the date command, the patterns would look for messages from whatever hour it is when the script runs, even if it's run near the end of an hour, something delays it's running, and it actually runs just after the start of a new hour.

You could also pass in the hour for which you want to search to be absolutely sure you get the right hour.

Whichever way you need to do things, hope this helps.
Later error in the log could be different than the past error message in the log.
"00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server"

rigor 04-07-2010 10:40 PM
Are you saying the message has no time stamp? It looks like this:

Code:
00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server
rather than like this:

Code:
[3/29/10 0:01:45:464 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.

dnaqvi 04-15-2010 11:08 AM
Quote:
Originally Posted by kakaka (Post 3928159)
Are you saying the message has no time stamp? It looks like this:

Code:
00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server
rather than like this:

Code:
[3/29/10 0:01:45:464 PDT] 00000093 LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://00.00.00.00:123.
yes


All times are GMT -5. The time now is 07:50 AM.