Need help in shell scripting

varadu1793 · 02-10-2013, 12:01 PM

Hello all,

My server is running RHEL5.5.
Another server in the network is pushing Call Detail Records ( CDRs ) into my server in a particular location. But unfortunately there are no sequence numbers in the CDRs.
I need to run a script so that as soon as the file is pushed to my server, a sequence number is added to the name and stored. The sequence number should follow a counter pattern.
I need some inputs in how this script is to be written.

Thanks in Advance
Ram

Snark1994 · 02-10-2013, 12:17 PM

A question may be urgent to you, but it's never urgent to us - so generally putting "Urgent" in your thread title is unhelpful. A better thread title would be along the lines of "Shell script to process files pushed from another server", or something similar. You might also have been better off posting this in the "Programming" forum (but don't go and post it in there again now, we'll be able to answer it here - if you want it moved, just click "Report" on one of the posts here and ask for the thread to be moved to the Programming forum).

In answer to your question: how do you know when a new file is pushed to the server? Does the pushed file overwrite the previously pushed file, or are they all in the same directory? How unique does the sequence number have to be? (i.e. do we restart from 1 every time the program is run, or every day, or every financial year, or never?) If you explain a bit more exactly what you're trying to achieve, it sounds very do-able: you just want to use 'mv' command, along with some process for storing/generating the sequence number (which will depend on your answer to the questions above).

Regards,

TB0ne · 02-10-2013, 01:43 PM

Quote:

Originally Posted by varadu1793

Hello all,
My server is running RHEL5.5.
Another server in the network is pushing Call Detail Records ( CDRs ) into my server in a particular location. But unfortunately there are no sequence numbers in the CDRs.

How is it getting 'pushed'? What protocol? Is this a network share, or are files being FTP'ed? Snark asked some questions too, and all of these things need to be considered.

Quote:

I need to run a script so that as soon as the file is pushed to my server, a sequence number is added to the name and stored. The sequence number should follow a counter pattern.

Ok, so that means that YOU need to write a script to do what you want. There are THOUSANDS of very easily-found scripting tutorials..there is even a link in my post signature. Since it's **URGENT** for you, you should start reading those tutorials right now, and following the examples.

Add to that, you have not really given ANY details about this. What are the CDR files called? What is the current naming location? Where are you getting the sequence number to start with? Can there be more than one with the same sequence number? What naming convention would you like to use for the output file? Moving them to the same system, or another??

We will be very happy to help you, but we are NOT going to write a script FOR YOU; please show some efforts of your own, and tell us where you're stuck.

varadu1793 · 02-10-2013, 10:01 PM

Hello All,

My files are getting pushed via SFTP from the network element to my server on location /tmp.

Right now the CDR naming convention has the date,day and time as the name of the CDR. But the customer needs sequence number added to the name. The sequence number can be incremented from 1 upwards.
I was thinking of getting this file to another location, modifying the name and moving it to the final destination.

Pls find below the script that i could perceive for such an action but unsuccessful

#!/bin/bash
cd /tmp/cdr_temp
counter=1
FILE=$1
if [ -f $FILE ];
then
rename * *_$counter *
counter=counter+1
mv * /tmp/billing/
else
fi

Your inputs will be highly appreciated.

Rgds
Ram

chrism01 · 02-11-2013, 02:06 AM

If they ARRIVE in date/time order, then 'ls -rt' will order them oldest-first => most-recent.

If ARRIVAL order cannot be guaranteed, then I'd use a more sophisticated approach by reading/parsing the filenames. Even then though, you'd have to allow a delay window to ensure you don't number them out of sequence.

varadu1793 · 02-11-2013, 02:15 AM

Hi Chris,

The CDRs are generated every 30 mins and as soon as it is generated, it is pushed to my server.
The CDR name right now is cdr200511151200. I want to add a sequence number after it. So new CDR should somewhat look like cdr200511151200 - sequence number.
In the script that I wrote, can you basically guide me as to how to accomplish the above. This is the first time I am scripting so your inputs will be highly appreciated.

Rgds
Ram

Snark1994 · 02-11-2013, 03:51 AM

In which case, probably your best bet is to have a script like the one you posted:

Code:

for f in /tmp/cdr_temp/* ; do
    #get the last used counter number
    counter=$(ls /tmp/billing | sed 's/.*\.\([^.]*\)$/\1/g' | sort | tail -n 1)
    #and increment it
    counter=$(($counter+1))
    mv $f /tmp/billing/$(basename $f)_$counter
done

Completely untested, and it won't deal nicely with the case where you get more than one file in /tmp/cdr_temp (i.e. it won't care about the order in which they arrived). You'd just have to put this in your crontab to run, say, every 15 minutes.

Regards,

allend · 02-11-2013, 05:07 AM

I suggest using the inotifywait command.

Code:

#!/bin/bash

# Script to watch for creation of files by in cdr_temp directory
#  and move to billing directory appending a counter.
#
# This script uses the -m option to inotifywait and should never exit.

TEMP_DIR=/tmp/cdr_temp/
BILLING_DIR=/tmp/billing/
COUNTER=1

inotifywait  -mq  --format '%f' \
  -e close_write $TEMP_DIR \
  | while read file; do
      mv $TEMP_DIR${file} $BILLING_DIR${file}$COUNTER && (( COUNTER+=1 ))
    done

varadu1793 · 03-25-2013, 02:56 AM

Hi Guys,

Please note the below code works perfectly

COUNTER_FILE='/etc/counter.conf'
TMP_LOCATION='/tmp/cdr_tmp'
FINAL_LOCATION='/tmp/billing'

for file in $(ls $TMP_LOCATION)
do
counter=$(cat $COUNTER_FILE)
mv $TMP_LOCATION/$file $FINAL_LOCATION/$file"_"$counter
counter=$((counter+1))
echo $counter > $COUNTER_FILE
done

Make a file counter.conf in /etc and you can put the start sequence number. We are putting the counter value there so that on reboots the value is preserved.

You can then run this as a cronjob every 5 mins.

Thanks everyone for all your inputs.

Rgds
Ram

colucix · 03-25-2013, 03:23 AM

Please use a descriptive title for your thread excluding words like 'urgent' or 'help'. Using a proper title makes it easier for members to help you. This thread has been reported for title modification. Please do not add replies that address the thread title.