LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 02-10-2013, 12:01 PM   #1
varadu1793
LQ Newbie
 
Registered: Dec 2012
Posts: 6

Rep: Reputation: Disabled
Need help in shell scripting


Hello all,

My server is running RHEL5.5.
Another server in the network is pushing Call Detail Records ( CDRs ) into my server in a particular location. But unfortunately there are no sequence numbers in the CDRs.
I need to run a script so that as soon as the file is pushed to my server, a sequence number is added to the name and stored. The sequence number should follow a counter pattern.
I need some inputs in how this script is to be written.

Thanks in Advance
Ram

Last edited by onebuck; 03-25-2013 at 10:17 AM. Reason: Remove unwarranted 'Urgent'
 
Old 02-10-2013, 12:17 PM   #2
Snark1994
Senior Member
 
Registered: Sep 2010
Location: Wales, UK
Distribution: Arch
Posts: 1,632
Blog Entries: 3

Rep: Reputation: 345Reputation: 345Reputation: 345Reputation: 345
A question may be urgent to you, but it's never urgent to us - so generally putting "Urgent" in your thread title is unhelpful. A better thread title would be along the lines of "Shell script to process files pushed from another server", or something similar. You might also have been better off posting this in the "Programming" forum (but don't go and post it in there again now, we'll be able to answer it here - if you want it moved, just click "Report" on one of the posts here and ask for the thread to be moved to the Programming forum).

In answer to your question: how do you know when a new file is pushed to the server? Does the pushed file overwrite the previously pushed file, or are they all in the same directory? How unique does the sequence number have to be? (i.e. do we restart from 1 every time the program is run, or every day, or every financial year, or never?) If you explain a bit more exactly what you're trying to achieve, it sounds very do-able: you just want to use 'mv' command, along with some process for storing/generating the sequence number (which will depend on your answer to the questions above).

Regards,
 
1 members found this post helpful.
Old 02-10-2013, 01:43 PM   #3
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,769

Rep: Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614Reputation: 2614
Quote:
Originally Posted by varadu1793 View Post
Hello all,
My server is running RHEL5.5.
Another server in the network is pushing Call Detail Records ( CDRs ) into my server in a particular location. But unfortunately there are no sequence numbers in the CDRs.
How is it getting 'pushed'? What protocol? Is this a network share, or are files being FTP'ed? Snark asked some questions too, and all of these things need to be considered.
Quote:
I need to run a script so that as soon as the file is pushed to my server, a sequence number is added to the name and stored. The sequence number should follow a counter pattern.
Ok, so that means that YOU need to write a script to do what you want. There are THOUSANDS of very easily-found scripting tutorials..there is even a link in my post signature. Since it's **URGENT** for you, you should start reading those tutorials right now, and following the examples.

Add to that, you have not really given ANY details about this. What are the CDR files called? What is the current naming location? Where are you getting the sequence number to start with? Can there be more than one with the same sequence number? What naming convention would you like to use for the output file? Moving them to the same system, or another??

We will be very happy to help you, but we are NOT going to write a script FOR YOU; please show some efforts of your own, and tell us where you're stuck.

Last edited by TB0ne; 02-10-2013 at 01:45 PM.
 
Old 02-10-2013, 10:01 PM   #4
varadu1793
LQ Newbie
 
Registered: Dec 2012
Posts: 6

Original Poster
Rep: Reputation: Disabled
Hello All,

My files are getting pushed via SFTP from the network element to my server on location /tmp.

Right now the CDR naming convention has the date,day and time as the name of the CDR. But the customer needs sequence number added to the name. The sequence number can be incremented from 1 upwards.
I was thinking of getting this file to another location, modifying the name and moving it to the final destination.

Pls find below the script that i could perceive for such an action but unsuccessful

#!/bin/bash
cd /tmp/cdr_temp
counter=1
FILE=$1
if [ -f $FILE ];
then
rename * *_$counter *
counter=counter+1
mv * /tmp/billing/
else
fi

Your inputs will be highly appreciated.

Rgds
Ram

Last edited by varadu1793; 02-11-2013 at 01:15 AM.
 
Old 02-11-2013, 02:06 AM   #5
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,289

Rep: Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034
If they ARRIVE in date/time order, then 'ls -rt' will order them oldest-first => most-recent.

If ARRIVAL order cannot be guaranteed, then I'd use a more sophisticated approach by reading/parsing the filenames. Even then though, you'd have to allow a delay window to ensure you don't number them out of sequence.
 
Old 02-11-2013, 02:15 AM   #6
varadu1793
LQ Newbie
 
Registered: Dec 2012
Posts: 6

Original Poster
Rep: Reputation: Disabled
Hi Chris,

The CDRs are generated every 30 mins and as soon as it is generated, it is pushed to my server.
The CDR name right now is cdr200511151200. I want to add a sequence number after it. So new CDR should somewhat look like cdr200511151200 - sequence number.
In the script that I wrote, can you basically guide me as to how to accomplish the above. This is the first time I am scripting so your inputs will be highly appreciated.

Rgds
Ram
 
Old 02-11-2013, 03:51 AM   #7
Snark1994
Senior Member
 
Registered: Sep 2010
Location: Wales, UK
Distribution: Arch
Posts: 1,632
Blog Entries: 3

Rep: Reputation: 345Reputation: 345Reputation: 345Reputation: 345
In which case, probably your best bet is to have a script like the one you posted:

Code:
for f in /tmp/cdr_temp/* ; do
    #get the last used counter number
    counter=$(ls /tmp/billing | sed 's/.*\.\([^.]*\)$/\1/g' | sort | tail -n 1)
    #and increment it
    counter=$(($counter+1))
    mv $f /tmp/billing/$(basename $f)_$counter
done
Completely untested, and it won't deal nicely with the case where you get more than one file in /tmp/cdr_temp (i.e. it won't care about the order in which they arrived). You'd just have to put this in your crontab to run, say, every 15 minutes.

Regards,
 
Old 02-11-2013, 05:07 AM   #8
allend
Senior Member
 
Registered: Oct 2003
Location: Melbourne
Distribution: Slackware-current
Posts: 3,464

Rep: Reputation: 852Reputation: 852Reputation: 852Reputation: 852Reputation: 852Reputation: 852Reputation: 852
I suggest using the inotifywait command.
Code:
#!/bin/bash

# Script to watch for creation of files by in cdr_temp directory
#  and move to billing directory appending a counter.
#
# This script uses the -m option to inotifywait and should never exit.

TEMP_DIR=/tmp/cdr_temp/
BILLING_DIR=/tmp/billing/
COUNTER=1

inotifywait  -mq  --format '%f' \
  -e close_write $TEMP_DIR \
  | while read file; do
      mv $TEMP_DIR${file} $BILLING_DIR${file}$COUNTER && (( COUNTER+=1 ))
    done
 
1 members found this post helpful.
Old 03-25-2013, 02:56 AM   #9
varadu1793
LQ Newbie
 
Registered: Dec 2012
Posts: 6

Original Poster
Rep: Reputation: Disabled
Hi Guys,

Please note the below code works perfectly

COUNTER_FILE='/etc/counter.conf'
TMP_LOCATION='/tmp/cdr_tmp'
FINAL_LOCATION='/tmp/billing'

for file in $(ls $TMP_LOCATION)
do
counter=$(cat $COUNTER_FILE)
mv $TMP_LOCATION/$file $FINAL_LOCATION/$file"_"$counter
counter=$((counter+1))
echo $counter > $COUNTER_FILE
done

Make a file counter.conf in /etc and you can put the start sequence number. We are putting the counter value there so that on reboots the value is preserved.

You can then run this as a cronjob every 5 mins.

Thanks everyone for all your inputs.

Rgds
Ram
 
Old 03-25-2013, 03:23 AM   #10
colucix
Moderator
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,508

Rep: Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957
Please use a descriptive title for your thread excluding words like 'urgent' or 'help'. Using a proper title makes it easier for members to help you. This thread has been reported for title modification. Please do not add replies that address the thread title.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
reading a file and sending mail by shell scripting..?? Plz help urgent..!! sukhdip Linux - Newbie 10 10-03-2011 02:16 AM
LXer: Terminal functions for shell scripting with Shell Curses LXer Syndicated Linux News 0 03-26-2008 11:50 PM
Shell Scripting: Getting a pid and killing it via a shell script topcat Programming 15 10-28-2007 02:14 AM
teaching shell scripting: cool scripting examples? fax8 Linux - General 1 04-20-2006 04:29 AM
shell interface vs shell scripting? I'm confused jcchenz Linux - Software 1 10-26-2005 03:32 PM


All times are GMT -5. The time now is 03:12 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration