LinuxQuestions.org - Some character adding up starting of log file?

- Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)

- - Some character adding up starting of log file? (https://www.linuxquestions.org/questions/linux-newbie-8/some-character-adding-up-starting-of-log-file-792511/)

Some character adding up starting of log file?

The logrotate is setup in one of the RHEL Machine and I could see surprising character at the top of the log file as:

^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@

upto certain lines everytime.
What could be the issue.

Exactly which file is this junk appearing in?

Its a customized log location in a file with extension .log

Generally what could be possiblilty of being ^@ added at the line 1
If I do :2 in commandline of log file it works fine.

Quote:

Originally Posted by your_shadow03 (Post 3882001)

Generally what could be possiblilty of being ^@ added at the line 1
If I do :2 in commandline of log file it works fine.

I'm not sure I understand what you've written there; could you perhaps show us these command-lines you are using? The one that adds the junk, and also the one which works correctly using ":2" whatever that is?

Also, maybe it isn't important, but this logfile, it is a log created by what application? Or is it a common/usual Linux system generated log? In other words, what process(s) are writing to the logfile?

Code:

#!/bin/bash

source ~/.bash_profile



export INST_NAME=testserver.com



if [ `ps -ef | grep -i run | grep -v grep | grep -i $INST_NAME | wc -l` = 0 ]; then



nohup sh /tools/jboss/jboss4/jboss-as/bin/run.sh -b testserver11.com -c testserver.com -D.data-dir=/fixer/hjs/config/server.name=testserver.com 2>&1 > /logs/hjs/fixer/jboss/testserver.com/testserver112.com &



sleep 3



tail -f /logs/hjs/fixer/jboss/testserver.com/testserver112.com



else

        echo "$INST_NAME Instance is already running"

fi

~

Just to inform you that logrotate has been working for the same.

OK, so would it be fair to suppose that this problem has little or nothing to do with `logrotate`, but
is very likely to have something to do with that script:

/tools/jboss/jboss4/jboss-as/bin/run.sh

either directly, or indirectly via a problem or typo with one of those arguments provided to the script:

-b testserver11.com -c testserver.com \
-Datg.dynamo.data-dir=/fixer/hjs/config/ATG-Data \
-Datg.dynamo.server.name=testserver.com

If you have double-and-triple checked that all these command-line arguments are correct, I would be then looking at the above mentioned script itself. Is there code inside the `run.sh` script which would generate literally the junk you get in the log? If so, examine the conditions required to cause the junk to be outputted.

Has this process ever worked correctly in the past? Without the crap in the logs? I.e. have you used this software before and it worked OK? If so, what's different this time?

Sasha

My logrotate file:

Code:

#####################  Source log file ######################

/logs/gm.log

{





# Show error if source log file is missing.

nomissingok



# log files are rotated when they grow bigger than size bytes.

size=1k



# log files are rotated 50 times before being removed.

rotate 30



# This is the number to use as the base for rotation.  Eg:  gm_log.0    gm.log.1

start 0



# Truncate the original log file in place after creating a copy.

copytruncate

My run.conf file:

Code:



CustomLog "| /usr/sbin/logrotate /mc.com/gm.log 60" common



JAVA_OPTS="-server -XX:PermSize=128m -XX:MaxPermSize=256m -XX:-PrintTenuringDistribution -XX:NewSize=256m -XX:MaxNewSize=384m -XX:+UseParallelGC -XX:SurvivorRatio=12 -Xmx1280m -Xms1280m -XX:ThreadStackSize=256 -XX:+DisableExplicitGC -verbose:gc -XX:+PrintGCDetails  -XX:+PrintTenuringDistribution -XX:ParallelGCThreads=2 -Xloggc:/mc.com/gm.log -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.lang.ClassLoader.allowArraySyntax=true"

the run.sh contains:

Code:

#!/bin/sh

### ====================================================================== ###

##                                                                          ##

##  JBoss Bootstrap Script                                                  ##

##                                                                          ##

### ====================================================================== ###



### $Id: run.sh 73862 2008-05-30 18:27:03Z mmoyses $ ###



DIRNAME=`dirname $0`

PROGNAME=`basename $0`

GREP="grep"



# Set conf if specified, else set to production

JBOSSCONF="production"

CONF_SPECIFIED=false



arg_count=1

eval SWITCH=\${$arg_count}

while [ ! -z "$SWITCH" ]

do



        if [ "$SWITCH" = "-c" ]; then

            eval JBOSSCONF=\$`expr $arg_count + 1`

            CONF_SPECIFIED=true

            break

        fi



        arg_count=`expr $arg_count + 1`

        eval SWITCH=\${$arg_count}

done



if [ x${CONF_SPECIFIED} = "xfalse" ]

then

  set -- "-c" ${JBOSSCONF} $@

fi



# Use the maximum available, or set MAX_FD != -1 to use that

MAX_FD="maximum"



#

# Helper to complain.

#

#

warn() {

    echo "${PROGNAME}: $*"

}



#

# Helper to puke.

#

die() {

    warn $*

    exit 1

}



# OS specific support (must be 'true' or 'false').

cygwin=false;

darwin=false;

linux=false;

case "`uname`" in

    CYGWIN*)

        cygwin=true

        ;;



    Darwin*)

        darwin=true

        ;;



    Linux)

        linux=true

        ;;

esac



# Read an optional running configuration file

if [ "x$RUN_CONF" = "x" ]; then



    if [ ! -z "$JBOSSCONF" ] && [ -f "$DIRNAME/../server/$JBOSSCONF/run.conf" ]; then

        RUN_CONF="$DIRNAME/../server/$JBOSSCONF/run.conf"

    else

        RUN_CONF="$DIRNAME/run.conf"

    fi

fi

if [ -r "$RUN_CONF" ]; then

    . "$RUN_CONF"



fi



# Force IPv4 on Linux systems since IPv6 doesn't work correctly with jdk5 and lower

if [ "$linux" = "true" ]; then

  JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"

fi



# For Cygwin, ensure paths are in UNIX format before anything is touched

if $cygwin ; then

    [ -n "$JBOSS_HOME" ] &&

        JBOSS_HOME=`cygpath --unix "$JBOSS_HOME"`

    [ -n "$JAVA_HOME" ] &&

        JAVA_HOME=`cygpath --unix "$JAVA_HOME"`

    [ -n "$JAVAC_JAR" ] &&

        JAVAC_JAR=`cygpath --unix "$JAVAC_JAR"`

fi



# Setup JBOSS_HOME

if [ "x$JBOSS_HOME" = "x" ]; then

    # get the full path (without any relative bits)

    JBOSS_HOME=`cd $DIRNAME/..; pwd`

fi

export JBOSS_HOME



# Increase the maximum file descriptors if we can

if [ "$cygwin" = "false" ]; then

    MAX_FD_LIMIT=`ulimit -H -n`

    if [ "$?" -eq 0 ]; then

        if [ "$MAX_FD" = "maximum" -o "$MAX_FD" = "max" ]; then

            # use the system max

            MAX_FD="$MAX_FD_LIMIT"

        fi



        ulimit -n $MAX_FD

        if [ "$?" -ne 0 ]; then

            warn "Could not set maximum file descriptor limit: $MAX_FD"

        fi

    else

        warn "Could not query system maximum file descriptor limit: $MAX_FD_LIMIT"

    fi

fi

# Setup the JVM

if [ "x$JAVA" = "x" ]; then

    if [ "x$JAVA_HOME" != "x" ]; then

        JAVA="$JAVA_HOME/bin/java"

    else

        JAVA="java"

    fi

fi



# Setup the classpath

runjar="$JBOSS_HOME/bin/run.jar"

if [ ! -f "$runjar" ]; then

    die "Missing required file: $runjar"

fi

JBOSS_BOOT_CLASSPATH="$runjar"



# Tomcat uses the JDT Compiler

# Only include tools.jar if someone wants to use the JDK instead.

# compatible distribution which JAVA_HOME points to

if [ "x$JAVAC_JAR" = "x" ]; then

    JAVAC_JAR_FILE="$JAVA_HOME/lib/tools.jar"

else

    JAVAC_JAR_FILE="$JAVAC_JAR"

fi

if [ ! -f "$JAVAC_JAR_FILE" ]; then

  # MacOSX does not have a seperate tools.jar

  if [ "$darwin" != "true" -a "x$JAVAC_JAR" != "x" ]; then

      warn "Missing file: JAVAC_JAR=$JAVAC_JAR"

      warn "Unexpected results may occur."

  fi

  JAVAC_JAR_FILE=

fi



if [ "x$JBOSS_CLASSPATH" = "x" ]; then

    JBOSS_CLASSPATH="$JBOSS_BOOT_CLASSPATH"

else

    JBOSS_CLASSPATH="$JBOSS_CLASSPATH:$JBOSS_BOOT_CLASSPATH"

fi

if [ "x$JAVAC_JAR_FILE" != "x" ]; then

    JBOSS_CLASSPATH="$JBOSS_CLASSPATH:$JAVAC_JAR_FILE"

fi



# If -server not set in JAVA_OPTS, set it, if supported

SERVER_SET=`echo $JAVA_OPTS | $GREP "\-server"`

if [ "x$SERVER_SET" = "x" ]; then



    # Check for SUN(tm) JVM w/ HotSpot support

    if [ "x$HAS_HOTSPOT" = "x" ]; then

        HAS_HOTSPOT=`"$JAVA" -version 2>&1 | $GREP -i HotSpot`

    fi



    # Enable -server if we have Hotspot, unless we can't

    if [ "x$HAS_HOTSPOT" != "x" ]; then

        # MacOS does not support -server flag

        if [ "$darwin" != "true" ]; then

            JAVA_OPTS="-server $JAVA_OPTS"

        fi

    fi

fi



# Setup JBosst Native library path

JBOSS_NATIVE_DIR="$JBOSS_HOME/bin/native"

if [ -d "$JBOSS_NATIVE_DIR" ]; then

    if $cygwin ; then

        export PATH="$JBOSS_NATIVE_DIR:$PATH"

        JBOSS_NATIVE_DIR=`cygpath --dos "$JBOSS_NATIVE_DIR"`

    fi

    if [ "x$LD_LIBRARY_PATH" = "x" ]; then

        LD_LIBRARY_PATH="$JBOSS_NATIVE_DIR"

    else

        LD_LIBRARY_PATH="$JBOSS_NATIVE_DIR:$LD_LIBRARY_PATH"

    fi

    export LD_LIBRARY_PATH

    if [ "x$JAVA_OPTS" = "x" ]; then

  if [ "x$JAVA_OPTS" = "x" ]; then

        JAVA_OPTS="-Djava.library.path=$JBOSS_NATIVE_DIR"

    else

        JAVA_OPTS="$JAVA_OPTS -Djava.library.path=$JBOSS_NATIVE_DIR"

    fi

fi



# Setup JBoss sepecific properties

JAVA_OPTS="-Dprogram.name=$PROGNAME $JAVA_OPTS"



# Setup the java endorsed dirs

JBOSS_ENDORSED_DIRS="$JBOSS_HOME/lib/endorsed"



# For Cygwin, switch paths to Windows format before running java

if $cygwin; then

    JBOSS_HOME=`cygpath --path --windows "$JBOSS_HOME"`

    JAVA_HOME=`cygpath --path --windows "$JAVA_HOME"`

    JBOSS_CLASSPATH=`cygpath --path --windows "$JBOSS_CLASSPATH"`

    JBOSS_ENDORSED_DIRS=`cygpath --path --windows "$JBOSS_ENDORSED_DIRS"`

fi



# Display our environment

echo "========================================================================="

echo ""

echo "  JBoss Bootstrap Environment"

echo ""

echo "  JBOSS_HOME: $JBOSS_HOME"

echo ""

echo "  JAVA: $JAVA"

echo ""

echo "  JAVA_OPTS: $JAVA_OPTS"

echo ""

echo "  CLASSPATH: $JBOSS_CLASSPATH"

echo ""

echo "========================================================================="

echo ""



while true; do

  if [ "x$LAUNCH_JBOSS_IN_BACKGROUND" = "x" ]; then



  if [ "x$LAUNCH_JBOSS_IN_BACKGROUND" = "x" ]; then

      # Execute the JVM in the foreground

      "$JAVA" $JAVA_OPTS \

        -Djava.endorsed.dirs="$JBOSS_ENDORSED_DIRS" \

        -classpath "$JBOSS_CLASSPATH" \

        org.jboss.Main "$@"

      JBOSS_STATUS=$?

  else

      # Execute the JVM in the background

      "$JAVA" $JAVA_OPTS \

        -Djava.endorsed.dirs="$JBOSS_ENDORSED_DIRS" \

        -classpath "$JBOSS_CLASSPATH" \

        org.jboss.Main "$@" &

      JBOSS_PID=$!

      # Trap common signals and relay them to the jboss process

      trap "kill -HUP  $JBOSS_PID" HUP

      trap "kill -TERM $JBOSS_PID" INT

      trap "kill -QUIT $JBOSS_PID" QUIT

      trap "kill -PIPE $JBOSS_PID" PIPE

      trap "kill -TERM $JBOSS_PID" TERM

      # Wait until the background process exits

      WAIT_STATUS=0

      while [ "$WAIT_STATUS" -ne 127 ]; do

        JBOSS_STATUS=$WAIT_STATUS

        wait $JBOSS_PID 2>/dev/null

        WAIT_STATUS=$?

      done

  fi

  # If restart doesn't work, check you are running JBossAS 4.0.4+

  #    http://jira.jboss.com/jira/browse/JBAS-2483

  # or the following if you're running Red Hat 7.0

  #    http://developer.java.sun.com/developer/bugParade/bugs/4465334.html

  if [ "$JBOSS_STATUS" -eq 10 ]; then

      echo "Restarting JBoss..."

  else

      exit $JBOSS_STATUS

  fi

done

Seems that run.sh is making the log file as binary.

The log created is a data file not normal log file.

How to delete the first line of a file without creatign the temporary file?

I have a log file which generates the junk character at the first line. I want to write a script to remove it without disturbing the file or creating the temporary file(

I tried this :

sed 1D test > tempfile.

But I dont want to create a temp file as jboss might be using that file.
Can it be done?

How about using the in-place flag

Code:

sed -i 1D test

Evo2.

Can I make it this way:

I need a script which will copy the log file(say A) in seperate new file(say B).
Remove the first line from B and copy it back to A.
Will it affect the log writing.
anyone who can help me with this script.

evo2,

Never used that command option -i.
Will it not disturb the log writing.
Are you sure?

Quote:

Originally Posted by your_shadow03 (Post 3882102)

Will it not disturb the log writing.

I don't know. Why not test it?

Quote:

Are you sure?

Yes. I'm sure I don't know ;-)

Evo2.

A question: Can you not wait until the process has finished writing to the file?

Evo2.

Hi,

About sed's -i option:

Quote:

-i[SUFFIX]
--in-place[=SUFFIX]
This option specifies that files are to be edited in-place. GNU sed does this by creating a temporary file and sending output to this file rather than to the standard output.1.

So it will break the pipe!

evo2 has a good point in post #6. Why can't you wait?

If you write line by line to the file then just omit the first line which is given by the process write the remaining lines.

Now thats me really confused !!

Code:

A question: Can you not wait until the process has finished writing to the file?

What it exactly means?

Hi,

At a certain point your program (the one that writes to the logfile) stops. Once that happens you can edit the logfile without having to worry about the program writing to it.

At some point the process will stop or be killed: eg. when you shut down your machine, or when the logs are rotated.

Evo2.

The Issue is I can't wait for that.
Let me elaborate. There is some junk character being added looks like ^@^@.......which fills up at first line only consuming more and more size..that makes upto 500-700MB size(surprisingly).So That makes my logs over loaded. Once i delete the first line then it freed 500-700 MB. Donno th reason why it does show.
but All I temporarily need to delete the first line.
So Cant wait..
Any Suggestion?

Is there any method I can include the sed script at postrotate directive in logrotate.conf?
Anyone who can help me with what postrotate can do ?

Hi,

To my knowledge there isn't a way to do this (lets hope I'm wrong about this, but I doubt it).

I would suggest, if at all possible, to have a look at the program that generates the logfile and make sure (add some code) the first line (the one with 'junk') isn't put in the logfile but discarded.

Hope this helps.

PS: Maybe your idea about logrotate could work. I'm not too familiar with it so maybe others can assist with that.

I tried addding this way:

Code:

#!/bin/bash

> any_temp_file

cp fix.log any_temp_file

sed -i 1D any_temp_file

cat any_temp_file > fix.log

Any idea if this works !!!

I think the best thing to do here is work out why you are getting that first line of rubbish in the log file. What program is it?

Evo2.

Hi,

Quote:

Originally Posted by your_shadow03 (Post 3882297)

I tried addding this way:

Code:

#!/bin/bash

> any_temp_file

cp fix.log any_temp_file

sed -i 1D any_temp_file

cat any_temp_file > fix.log

Any idea if this works !!!

Strickly speaking: No, it won't (you will probably lose some logged data).

But.....

Depending on the frequency of the logging it _might_ work. If nothing is logged while you run the above script it will work, if on the other hand the program does log one or more entries while you run the script, those entries will not show up in the log.

Hope this helps.

I have pasted run.sh above. Anyone who have hands on Jboss and could solve the issue.

1. Give me the output of file <log_filename>.
2. Who is writitng this log. (Any java code) ?

This seems to me some kind of binary data been written in to your log file (A mix of ascii and binary I guess). The best way is to identify the Root cause.

Code:

file testfile.log



testfile: data

Code:

-rw-r--r-- 1 weadmin admingrp 3922 Mar 2 06:56 testfile.log

@your_shadow03: This is a binary formatted file, _not_ a text file.

You still haven't mentioned what program creates/writes to this log file, but it is done in a binary format.

My run.conf says:

Code:

JAVA_OPTS="-server -XX:PermSize=128m -XX:MaxPermSize=256m -XX:-PrintTenuringDistribution -XX:NewSize=256m -XX:MaxNewSize=384m -XX:+UseParallelGC -XX:SurvivorRatio=12 -Xmx1280m -Xms1280m -XX:ThreadStackSize=256 -XX:+DisableExplicitGC -verbose:gc -XX:+PrintGCDetails  -XX:+PrintTenuringDistribution -XX:ParallelGCThreads=2 -Xloggc:/project/gm.log -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.lang.ClassLoader.allowArraySyntax=true"

Code:

JAVA_OPTS="-server -XX:PermSize=128m -XX:MaxPermSize=256m -XX:-PrintTenuringDistribution -XX:NewSize=256m -XX:MaxNewSize=384m -XX:+UseParallelGC -XX:SurvivorRatio=12 -Xmx1280m -Xms1280m -XX:ThreadStackSize=256 -XX:+DisableExplicitGC -verbose:gc -XX:+PrintGCDetails  -XX:+PrintTenuringDistribution -XX:ParallelGCThreads=2 -Xloggc:/project/gm.log -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.lang.ClassLoader.allowArraySyntax=true"

Quote:

-Xloggc:file
Report on each garbage collection event, as with -verbose:gc, but log this data to file. In addition to the information -verbose:gc gives, each reported event will be preceeded by the time (in seconds) since the first garbage-collection event.
Always use a local file system for storage of this file to avoid stalling the JVM due to network latency. The file may be truncated in the case of a full file system and logging will continue on the truncated file. This option overrides -verbose:gc if both are given on the command line.

Please check you are using both the options. This probably is not the cause for the binary data in the file. But it is there so I highlighted it.

Secondly, I am still not sure how can you be so sure that these junk character will be there on the first line. For me these are gc logs and can be at any place.

Do you find any exception just above or below these junk character in the log file?

You can try invoking this java command on the command line to see what could be the probable exeception that is dumping these chars in the gc log file.

Hi,

I'm not a java man, but doesn't this: -Xloggc:/project/gm.log mean you are logging the garbage collector? I do believe you need a special viewer to 'see' that data.

A quick search comes up with GCView (freshmeat).

Hope this helps.

Can anyone confirm the same as I am not sure if that could be the reason.
What entry I should make to the file?

your_shadow03,

I've yet again merged two of your threads on the same subject; you've
been warned for double-/ multi-posts so many times it's not even funny.

Next double post of yours I'll give you an out-time of a few weeks.

Cheers,
Tink

Sorry Moderator,

The Two Thread are different but the result will the same.
Thats what made me ask in different respect.
Anyway, I am trying to get help from experts in two different ways.

anyway, will take care for next time.

Guys !!!
What my main aim is to remove this first line with junk character.
I did attempted removing the line through script but couldnt make it happen.
It does disturb the logs provisioning.
But ASAP I copy it temporarily to a file , make changes and copy it back those characters again appear.
Seems that something has to be done with run.sh (==<< Main culprit).

Hi,

I've re-read most of the merged thread and see one thing that I cannot place:

You (the OP) mention that the following line is in your run.conf file:

CustomLog "| /usr/sbin/logrotate /mc.com/gm.log 60" common

Should that be there? To my knowledge that entry isn't jboss specific (looks more like apache/tomcat).

Hope this helps.

I dont think it should affect the logs being provisioned as binary format.

Hi,

Have you actually tried to remove it.

Well,, I am not in corporate VPN..
Will surely give it try and let you know the status.
You can suggest me the other options too..if thats in your mind.
I have put the same query to Jboss Community too.
Lets see if it could be helpful.

I tried it now..
It's not provisioning this time since that is only entry in run.conf.
Removing it from the file will not provision the logs.

Hi,

It is the only entry?? I doubt it. This was provided by you in post #8 as being the content of run.conf:

Quote:

CustomLog "| /usr/sbin/logrotate /mc.com/gm.log 60" common

JAVA_OPTS="-server -XX:PermSize=128m -XX:MaxPermSize=256m -XX:-PrintTenuringDistribution -XX:NewSize=256m -XX:MaxNewSize=384m -XX:+UseParallelGC -XX:SurvivorRatio=12 -Xmx1280m -Xms1280m -XX:ThreadStackSize=256 -XX:+DisableExplicitGC -verbose:gc -XX:+PrintGCDetails -XX:+PrintTenuringDistribution -XX:ParallelGCThreads=2 -Xloggc:/mc.com/gm.log -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.lang.ClassLoader.allowArraySyntax=true"

- The first entry is not jboss/java specific (as stated before).
- Both entries seem to point to gm.log (the -Xloggc:/mc.com/gm.log in the JAVA_OPTS line).