LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 11-11-2021, 12:14 PM   #1
LinuxRSA
LQ Newbie
 
Registered: Apr 2015
Location: South Africa
Posts: 26

Rep: Reputation: Disabled
Display Last 5 Minutes of Log file Only


Hi All

I have scripted the below, it checks for HTTP Codes within the NGINX logs.

This is the script:

#!/bin/bash
FILE=/var/log/nginx/access.log
if cat /var/log/nginx/access.log | awk '{print $9}' | grep '400\|401\|403\|404\|409\|500\|501\|502\|503\|504'; then
echo "Nginx Issues"
else
echo "Nginx Healthy"
fi

This is the output:

server:~$ vi testngnix.sh
server:~$
server:~$ cat testngnix.sh
#!/bin/bash
FILE=/var/log/nginx/access.log
if cat /var/log/nginx/access.log | awk '{print $9}' | grep '400\|401\|403\|404\|409\|500\|501\|502\|503\|504'; then
echo "Nginx Issues"
else
echo "Nginx Healthy"
fi
server:~$
server:~$ ./testngnix.sh
502
502
502
502
502
502
502
502
Nginx Issues
server:~$

The challenge I'm having is this script displays the Entire file.

I wish to display the Last 5 minutes of the /var/log/nginx/access.log only.

How can this be achieved ?

Thanks

Last edited by LinuxRSA; 11-11-2021 at 12:33 PM.
 
Old 11-11-2021, 12:50 PM   #2
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 24,296

Rep: Reputation: 7140Reputation: 7140Reputation: 7140Reputation: 7140Reputation: 7140Reputation: 7140Reputation: 7140Reputation: 7140Reputation: 7140Reputation: 7140Reputation: 7140
Quote:
Originally Posted by LinuxRSA View Post
Hi All
I have scripted the below, it checks for HTTP Codes within the NGINX logs. This is the script:
Code:
#!/bin/bash
FILE=/var/log/nginx/access.log
if cat /var/log/nginx/access.log | awk '{print $9}' | grep '400\|401\|403\|404\|409\|500\|501\|502\|503\|504'; then
 echo "Nginx Issues"
else
 echo "Nginx Healthy"
fi
This is the output:
Code:
server:~$ vi testngnix.sh
server:~$
server:~$ cat testngnix.sh
#!/bin/bash
FILE=/var/log/nginx/access.log
if cat /var/log/nginx/access.log | awk '{print $9}' | grep '400\|401\|403\|404\|409\|500\|501\|502\|503\|504'; then
  echo "Nginx Issues"
  else
  echo "Nginx Healthy"
 fi
server:~$
server:~$ ./testngnix.sh
502
502
502
502
502
502
502
502
Nginx Issues
server:~$
The challenge I'm having is this script displays the Entire file. I wish to display the Last 5 minutes of the /var/log/nginx/access.log only. How can this be achieved ?
Put your code in CODE tags, please. And you achieve this by changing your script, obviously enough. You've been an 'administrator' for at least six years now, and have been posting here at least that long. You should know that you need to post a complete question, such as what the log file looks like going IN to the script, and what you've done to actually try to do what you're after. There are many ways to accomplish this, depending on the input data or what your needs are...could be as simple as running the script every five minutes, and having the last part of the script do a "cat /dev/null > /your/log/file", to zero it out. That way, there is only EVER 5 minutes of data in it, but that's not a good way to do this (but it will work).

Post a sample of the input data (again, in CODE tags), and show us what you've done to try to parse it, and tell us what your needs/goals are for the log files in general.
 
1 members found this post helpful.
Old 11-11-2021, 12:54 PM   #3
shruggy
Senior Member
 
Registered: Mar 2020
Posts: 3,037

Rep: Reputation: Disabled
Gawk has time functions, and the 4th field of access.log (in default "combined" log format) should be a timestamp.

Besides, this
Quote:
Originally Posted by LinuxRSA View Post
Code:
#!/bin/bash
FILE=/var/log/nginx/access.log
if cat /var/log/nginx/access.log | awk '{print $9}' | grep '400\|401\|403\|404\|409\|500\|501\|502\|503\|504'; then
 echo "Nginx Issues"
else
 echo "Nginx Healthy"
fi
could be rewritten as
Code:
#!/bin/sh
FILE=/var/log/nginx/access.log
awk -vr=Healthy '
  $9~/^([45]0[0-4]|409)$/ { print $9; r="Issues" }
  END { print "Nginx", r }' "$FILE"
See Pitfall: shorten pipelines

Hint. It will be easier to evaluate timestamps if you apply a custom format to the access log, e.g. replace $time_local with $time_iso8601 or complement it with $msec.

Last edited by shruggy; 11-11-2021 at 06:47 PM.
 
Old 11-12-2021, 12:43 AM   #4
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 7.7 (?), Centos 8.1
Posts: 17,925

Rep: Reputation: 2631Reputation: 2631Reputation: 2631Reputation: 2631Reputation: 2631Reputation: 2631Reputation: 2631Reputation: 2631Reputation: 2631Reputation: 2631Reputation: 2631
As hinted at here https://en.wikipedia.org/wiki/List_of_HTTP_status_codes, I'd just test the first digit, thereby allowing for new codes to be created and a bit easier to read...
 
Old 11-12-2021, 04:44 AM   #5
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,971
Blog Entries: 3

Rep: Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102Reputation: 3102
A very big obstacle here is the relative time requirement of finding entries from five minutes ago or less. I'm not sure that gAWK can be made to do that in any convenient way. Perl, on the other hand, does have Time::Piece and many other CPAN modules for managing time. An easy way might be to just subtract 300 seconds, after parsing that horrible¹ "combined" log format which I presume is a left over from the NCSA HTTP server.

Code:
#!/usr/bin/perl                                                                 

use Time::Piece;

# LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined                                                                            
# http://httpd.apache.org/docs/2.2/mod/mod_log_config.html#formats              
# http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format             

my $line = '127.0.0.1 - - [12/Nov/2021:10:32:04 +0300] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36"';

print qq($line\n);

# use named capture groups
$line =~ /^                                                                     
      (?<host>\S+)\s+                                                           
      (?<remotelogname>\S+)\s+                                                  
      (?<remoteuser>\S+)\s+                                                     
      \[(?<datetime>[^\]]+)\]\s+                                                
      "(?<request>[^"]+)"\s+                                                    
      (?<status>[^\]]+)\s+                                                      
      (?<bytes>[^\]]+)\s+                                                       
      "(?<referer>[^"]+)"\s+                                                    
      "(?<useragent>[^"]+)"                                                     
      /x;

my $datetime = $+{datetime};

my $dt = Time::Piece->strptime($datetime, "%d/%b/%Y:%H:%M:%S %z");
my $lt = Time::Piece->localtime();

print qq(datetime = $dt\n);
print qq(localtime = $lt\n);
print qq(5 min ago = ), $lt-300, qq(\n);

exit(0);
See "man Time::Piece" and "man perlre"

There might be edge cases with that around switching from Daylight Saving Time to Standard Time and back again. If that is a worry, then maybe look at Date::Calc instead. However, keep in mind Parkinson's Law.




¹ Another way is to use a custom log format with tab-delimited values so split() can be used to parse the log file lines.
 
Old 11-12-2021, 05:27 AM   #6
shruggy
Senior Member
 
Registered: Mar 2020
Posts: 3,037

Rep: Reputation: Disabled
Quote:
Originally Posted by Turbocapitalist View Post
I'm not sure that gAWK can be made to do that in any convenient way.
Assuming we have the default access log format:
Code:
#!/bin/sh
FILE=/var/log/nginx/access.log
gawk -vr=Healthy '
  BEGIN{
    m["Jan"]=1;m["Feb"]=2;m["Mar"]=3;m["Apr"]=4;  m["May"]=5;  m["Jun"]=6
    m["Jul"]=7;m["Aug"]=8;m["Sep"]=9;m["Oct"]=10; m["Nov"]=11; m["Dec"]=12
  }
  $9~/^[45]/{
    split($4,d,"/")
    split(d[3],t,":")
    d[1]=substr(d[1],2)
    ds=sprintf("%d %d %d %d %d %d",t[1],m[d[2]],d[1],t[2],t[3],t[4])
    ts=mktime(ds)
    if ( systime()-ts < 300 ) print $9
    r="Issues"
  }
  END{print "Nginx",r}' "$FILE"
If dateutils are installed, month names could be converted to numbers on the fly (on Debian-based distros, the command name is dateutils.dconv rather than dateconv):
Code:
#!/bin/sh
FILE=/var/log/nginx/access.log
# dateconv (Fedora), dateutils.dconv (Debian)
gawk -vr=Healthy '
  $9~/^[45]/{
    split($4,d,"/")
    split(d[3],t,":")
    d[1]=substr(d[1],2)
    dc="dateconv -i%b -f%m " d[2]
    dc | getline m
    close(dc)
    ds=sprintf("%d %d %d %d %d %d",t[1],m,d[1],t[2],t[3],t[4])
    ts=mktime(ds)
    if ( systime()-ts < 300 ) print $9
    r="Issues"
  }
  END{print "Nginx",r}' "$FILE"
As I said above, changing the log format to provide $time_iso8601 or $msec would simplify the things quite a bit.

Last edited by shruggy; 11-12-2021 at 05:28 AM.
 
1 members found this post helpful.
Old 11-12-2021, 10:41 AM   #7
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 24,296

Rep: Reputation: 7140Reputation: 7140Reputation: 7140Reputation: 7140Reputation: 7140Reputation: 7140Reputation: 7140Reputation: 7140Reputation: 7140Reputation: 7140Reputation: 7140
Correct to all, and Time::Piece is a great tool for such things. But the OP never posted back with a sample of whatever data they're using...speculation until they answer.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
How to log internal-sftp chroot jailed users access log to /var/log/sftp.log file LittleMaster Linux - Server 0 09-04-2018 04:45 PM
Get first day of last month and last day of last month in bash xowl Linux - Software 18 02-09-2017 10:49 AM
how to disable "last login log" & disable "last login message" when start login. hocheetiong Linux - Newbie 4 02-08-2011 06:35 AM
Show last 10 minutes of the log frater Linux - Software 6 12-16-2010 06:03 PM
Selecting last 60 minutes from log files WojtekO Linux - Software 5 04-10-2009 03:29 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 11:38 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration