Issue with grep command

nextStep · 08-15-2018, 04:54 AM

Hi All,

I am new to shell script. My requirement is to list user information from remote log files. For that purpose wrote shell script as below(For security reason masked real value)

Code

ssh -i /home/user/.ssh/id_rsa user@hostname zgrep -i -B8 "Decision from API: DECLINED" /path1/pah2/path3/logfile.gz|zgrep "API detection is enabled"

The issue is the above command, if executed from remote terminal will provide the result as expected, but if we run as shell script from the host server the output varies.

This i think grep stop searches after finding the first pattern("Decision from API") and returns the record which is having logs other than "Declined" also.

Could you please advise how to fix.

JJJCR · 08-15-2018, 05:30 AM

Quote:

Originally Posted by nextStep

Hi All,

I am new to shell script. My requirement is to list user information from remote log files. For that purpose wrote shell script as below(For security reason masked real value)

Code

ssh -i /home/user/.ssh/id_rsa user@hostname zgrep -i -B8 "Decision from API: DECLINED" /path1/pah2/path3/logfile.gz|zgrep "API detection is enabled"

The issue is the above command, if executed from remote terminal will provide the result as expected, but if we run as shell script from the host server the output varies.

This i think grep stop searches after finding the first pattern("Decision from API") and returns the record which is having logs other than "Declined" also.

Could you please advise how to fix.

Remote and logon user has the same account privileges?

Are they logon also to the same profile with remote user?

Turbocapitalist · 08-15-2018, 05:37 AM

Welcome to the forum. It helps to put [code] [/code] tags around actual code to make it more readable. The second zgrep should be a regular grep since it is dealing with regular text output from the first one:

Code:

ssh -i /home/user/.ssh/id_rsa user@hostname 'zgrep -i -h -B8 "Decision from API: DECLINED" /path1/pah2/path3/logfile.gz' \
        | grep "API detection is enabled"

The zgrep runs on the remote computer. The grep runs on the local one.

The -h option might also help, since you won't need the file name included. See "man grep"

MadeInGermany · 08-15-2018, 05:45 AM

The ssh command is evaluated twice, once on the calling host and once on the remote host.
The calling host removes the "quotes", and they are missing on the remote site.
Some work-arounds:

Code:

ssh -i /home/user/.ssh/id_rsa user@hostname zgrep -i -B8 \"Decision from API: DECLINED\" /path1/pah2/path3/logfile.gz | zgrep "API detection is enabled"

The calling host replaces the escaped " by a ". The | zgrep ... runs on the calling host.

Code:

ssh -i /home/user/.ssh/id_rsa user@hostname 'zgrep -i -B8 "Decision from API: DECLINED" /path1/pah2/path3/logfile.gz' | zgrep "API detection is enabled"

The calling host sees the 'string in ticks' and removes the ticks. The | zgrep ... runs on the calling host.

Code:

ssh -i /home/user/.ssh/id_rsa user@hostname 'zgrep -i -B8 "Decision from API: DECLINED" /path1/pah2/path3/logfile.gz | zgrep "API detection is enabled"'

The calling host sees the 'string in ticks' and removes the ticks. The | zgrep ... runs on the remote host.

nextStep · 08-15-2018, 05:58 AM

Quote:

Originally Posted by Turbocapitalist

Welcome to the forum. It helps to put [code] [/code] tags around actual code to make it more readable. The second zgrep should be a regular grep since it is dealing with regular text output from the first one:

Code:

ssh -i /home/user/.ssh/id_rsa user@hostname 'zgrep -i -h -B8 "Decision from API: DECLINED" /path1/pah2/path3/logfile.gz' \
        | grep "API detection is enabled"

The zgrep runs on the remote computer. The grep runs on the local one.

The -h option might also help, since you won't need the file name included. See "man grep"

Thanks a lot, the above approach worked.

nextStep · 08-15-2018, 06:02 AM

Quote:

Originally Posted by MadeInGermany

The ssh command is evaluated twice, once on the calling host and once on the remote host.
The calling host removes the "quotes", and they are missing on the remote site.
Some work-arounds:

Code:

ssh -i /home/user/.ssh/id_rsa user@hostname zgrep -i -B8 \"Decision from API: DECLINED\" /path1/pah2/path3/logfile.gz | zgrep "API detection is enabled"

The calling host replaces the escaped " by a ". The | zgrep ... runs on the calling host.

Code:

ssh -i /home/user/.ssh/id_rsa user@hostname 'zgrep -i -B8 "Decision from API: DECLINED" /path1/pah2/path3/logfile.gz' | zgrep "API detection is enabled"

The calling host sees the 'string in ticks' and removes the ticks. The | zgrep ... runs on the calling host.

Code:

ssh -i /home/user/.ssh/id_rsa user@hostname 'zgrep -i -B8 "Decision from API: DECLINED" /path1/pah2/path3/logfile.gz | zgrep "API detection is enabled"'

The calling host sees the 'string in ticks' and removes the ticks. The | zgrep ... runs on the remote host.

Thanks for the useful tips.This also worked.

nextStep · 08-15-2018, 06:07 AM

Quote:

Originally Posted by nextStep

Thanks a lot, the above approach worked.

One query :
Why the single quote put before the zgrep and why \ given before the pipe symbol.

'zgrep -i -h -B8 "Decision from API: DECLINED" /path1/path2/path3/logfile.gz' \

Turbocapitalist · 08-15-2018, 06:10 AM

If the last character in a line is a backslash \ then it tells the shell interpreter that things are continued on the next line. Sometimes it is easier to read when separate instructions are on separate lines even if they are piped from one to the next.

The single quote just groups things for sending to the remote server. Everything else after that is done locally.

MadeInGermany · 08-15-2018, 06:57 AM

Usually I avoid a \ at line ends, because copy/paste sometimes adds a space character at the very end.
Instead I break a long line into logical blocks, like this

Code:

ssh -i /home/user/.ssh/id_rsa user@hostname '
  zgrep -i -B8 "Decision from API: DECLINED" /path1/pah2/path3/logfile.gz | grep "API detection is enabled"
'

The remote shell

sees the two empty lines and ignores them.
sees the indented command, and this has no impact.