[SOLVED] BASH : how to sort bunch of .log files ?

nartotik · 11-06-2019, 12:06 PM

hi guys.
im new to bash scripting so i need your help.
i got an assignment that im having trouble to deal with.

i got a bunch of .log files that some of them are "corrupt"
example for a "corrupt" log file:
2019-08-22T13:33:44.123456789 Hello
World
2020-01-01T11:22:33.123456789 late

i need to write a script that sort the .log files in a given path (also in the sub directories of the given path) and keep the content of the log file.

example on 3 log file:
1.log
2019-08-22T13:33:44.123456789 Hello
World
2020-01-01T11:22:33.123456789 late
2.log
2019-09-44T13:44:21.987654321 Simple line
3.log
2019-08-22T13:33:44.123456789 Hello
World 2
2020-01-01T11:22:33.123456789 late 2

the output needs to be:
‫‪2019-08-22T13:33:44.123456789‬‬ ‫‪Hello‬‬
‫‪World‬‬
‫‪2019-08-22T13:33:44.123456789‬‬ ‫‪Hello‬‬
‫‪World‬‬ ‫‪2‬‬
‫‪2019-09-44T13:44:21.987654321‬‬ ‫‪Simple‬‬ ‫‪line‬‬
‫‪2020-01-01T11:22:33.123456789‬‬ ‫‪late‬‬
‫‪2020-01-01T11:22:33.123456789‬‬ ‫‪late‬‬ ‫‪2‬‬

Code:

my script so far:
#!/bin/bash
path=$1
if cd $path;then
	output="$(find ${dir} -name "*.log" | xargs cat | tr -d '\0' | sort -n -k1,1 -k2,2 )"
	echo "${output}"
	(exit 0)
else
	(exit 2);
fi

thanks in advance for all the helpers!!

pan64 · 11-06-2019, 12:20 PM

1. please use [code]code tags[/code] to post code (and keep formatting).
2. please use shellcheck to check your script
3. I do not understand what is the difference between the good and corrupt files.

nartotik · 11-06-2019, 12:30 PM

Quote:

Originally Posted by pan64

1. please use [code]code tags[/code] to post code (and keep formatting).
2. please use shellcheck to check your script
3. I do not understand what is the difference between the good and corrupt files.

1. edited
2. used it , i dont have syntax errors, i dont understand how can i do it.
3. the difference is that good log files dosent contains random words in some lines so i cant use sort command on the merged log files as expected , if im using it it pushes the random words to the end

Firerat · 11-06-2019, 12:42 PM

well

I have no idea what your script is trying to do

Code:

#!/bin/bash
path=$1
# change dir, so for so good ( not *needed* but another story )
if cd $path;then
	output="$(find ${dir} -name "*.log" | xargs cat | tr -d '\0' | sort -n -k1,1 -k2,2 )"
        # you are concatenating all the logs, trimming and sorting and capturing
        # the contents into a variable 
	echo "${output}"
        # no you have printed the contents of all the logs to screen
	(exit 0)
        # started a subshell and exited that subshell
else
	(exit 2);
        # started a subshell and exited that subshell  
fi

basically it does nothing useful

please try to explain the problem better

I'm with @pan64 in not understanding the good/bad logs

in the meantime https://mywiki.wooledge.org/BashGuide
that should fix the many bad habits you demonstrate in your script

Firerat · 11-06-2019, 01:42 PM

had a little think

if the log is delimited by something other than a "space" it will need to be adapted

Code:

#!/bin/bash
path="$1"
while read line
do
  [[ ${line} =~ ^[0-9]{4}(-[0-9]{2}){2} ]] \
    && printf "\n" \
    || printf " "
  printf "%s" "${line}"
done < <( cat ${path%/}/*.log ) | sort -k1

this begins to add some checking

Code:

#!/bin/bash
path="$1"

errorreport () {
Ecode=$?
case $Ecode in
    0)  
        :
        ;;
    4)
        echo "no log supplied"
        ;;
    5)
        echo "$path is not a directory"
        ;;
    *)
        echo "unknown error"
esac
exit $Ecode
}
trap errorreport EXIT

[[ -z $path ]] && exit 4
[[ -d $path ]] || exit 5

while read line
do
  [[ ${line} =~ ^[0-9]{4}(-[0-9]{2}){2} ]] \
    && printf "\n" \
    || printf " "
  printf "%s" "${line}"
done < <( cat ${path%/}/*.log ) | sort -k1

edit,
the output is

Code:

2019-08-22T13:33:44.123456789 Hello World
2019-08-22T13:33:44.123456789 Hello World 2
2019-09-44T13:44:21.987654321 Simple line
2020-01-01T11:22:33.123456789 late
2020-01-01T11:22:33.123456789 late 2

Edit 2
if you need help with the field separators
either upload a raw sample of your logs
or the output of

Code:

cat -A sample.log

basically we can use the read to load each line into an array with its elements based on the field separator.
We can then use printf to "re-print" the lines with correct field separator