LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 05-27-2009, 09:11 PM   #1
dfresh4130
LQ Newbie
 
Registered: May 2009
Posts: 6

Rep: Reputation: 0
Question Replacing text in files without using sed


I've run into an issue here. I have a perl script that takes some logs on a daily basis and creates a report from them. Recently the format of the logs was changed and that's causing the report to give wrong data. I've been trying to have the log files edited to remove the new data causing problems and I've been successful with the sed command. However, whenever I use the perl script on those new logs it gives me out of memory errors. Absolutely no idea why because when I manually edit them it works fine. So whenever I've used sed to touch those logs the perl script blows up. Below is a snippet of the log formats. I need to get rid of the jds-JDS: and jds values on each line. Does anyone know how to do this with another command? I tried cut and tr, but neither of them work the way I needed it to.

Tue May 26 00:00:00 CDT 2009|352|getDetailedUserProfileByUsername|jds
Tue May 26 00:00:02 CDT 2009|896|querySubscriberUnbilledUsageSummaryBtRequestBT|jds
Tue May 26 00:00:02 CDT 2009|1139|querySubscriberUnbilledUsageSummaryBtRequestBT|jds
Tue May 26 00:00:02 CDT 2009|353|getSubsInfo|jds-JDS:124331400223813
Tue May 26 00:00:02 CDT 2009|753|getSubsInfo|jds-JDS:124331400200410
Tue May 26 00:00:03 CDT 2009|1208|getCustSecurityInfo|jds-JDS:124331400201512
Tue May 26 00:00:03 CDT 2009|1301|getBanInfo|jds-JDS:124331400201411
Tue May 26 00:00:03 CDT 2009|734|getSubsInfo|jds-JDS:124331400297914
Tue May 26 00:00:03 CDT 2009|219|authenticate|jds
Tue May 26 00:00:03 CDT 2009|294|getDetailedUserProfileByUsername|jds
Tue May 26 00:00:04 CDT 2009|207|getSubsInfo|jds-JDS:124331400390717
Tue May 26 00:00:04 CDT 2009|460|getSubsInfo|jds-JDS:124331400378816
Tue May 26 00:00:05 CDT 2009|494|getSubsInfo|jds-JDS:124331400512718
Tue May 26 00:00:05 CDT 2009|196|getBanInfo|jds-JDS:124331400567820
Tue May 26 00:00:05 CDT 2009|426|getSubsInfo|jds-JDS:124331400553219
Tue May 26 00:00:06 CDT 2009|2267|getBanInfo|jds-JDS:124331400374615
Tue May 26 00:00:06 CDT 2009|519|getBanInfo|jds-JDS:124331400573622
Tue May 26 00:00:06 CDT 2009|556|getSubsInfo|jds-JDS:124331400573621
Tue May 26 00:00:06 CDT 2009|219|getBanInfo|jds-JDS:124331400608724
Tue May 26 00:00:06 CDT 2009|685|getCustSecurityInfo|jds-JDS:124331400573623
Tue May 26 00:00:06 CDT 2009|267|getBanInfo|jds-JDS:124331400622726
Tue May 26 00:00:06 CDT 2009|566|getCustSecurityInfo|jds-JDS:124331400622827
 
Old 05-27-2009, 09:18 PM   #2
normscherer
Member
 
Registered: Sep 2005
Location: On the road
Distribution: Ubuntu 8.10
Posts: 40

Rep: Reputation: 15
Why don't you just fix the perl script to accept the new input format?
 
Old 05-27-2009, 09:26 PM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,354

Rep: Reputation: 1042Reputation: 1042Reputation: 1042Reputation: 1042Reputation: 1042Reputation: 1042Reputation: 1042Reputation: 1042
Or better, make it insensitive to format changes. You could also do the mangling in that perl script prior to the current processing - anything you can do in sed perl will handle.
 
Old 05-27-2009, 09:35 PM   #4
dfresh4130
LQ Newbie
 
Registered: May 2009
Posts: 6

Original Poster
Rep: Reputation: 0
If I could fix the perl script I would. I'm no perl expert and the script is a remnant left over from a previous coworker who was a perl guru and his programming skills were cryptic to say the least.
 
Old 05-27-2009, 09:51 PM   #5
billymayday
Guru
 
Registered: Mar 2006
Location: Sydney, Australia
Distribution: Fedora, CentOS, OpenSuse, Slack, Gentoo, Debian, Arch, PCBSD
Posts: 6,678

Rep: Reputation: 122Reputation: 122
Could you show a line before and after manual edit, before and after sed edit and the sed command you use?
 
Old 05-27-2009, 09:52 PM   #6
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,696
Blog Entries: 5

Rep: Reputation: 241Reputation: 241Reputation: 241
Quote:
Originally Posted by dfresh4130 View Post
If I could fix the perl script I would. I'm no perl expert
don't really need expert to do replacement of strings with Perl. Since you are paid (is it not??) to do your job, its only natural you start reading up on Perl because eventually you are going to maintain those Perl scripts your predecessor has written.
 
Old 05-27-2009, 10:11 PM   #7
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,354

Rep: Reputation: 1042Reputation: 1042Reputation: 1042Reputation: 1042Reputation: 1042Reputation: 1042Reputation: 1042Reputation: 1042
That may be true, but I can understand the OP not wanting to screw around with something that worked.
Mangling the input should work - I'm also interested to what could cause this from sed usage.
 
Old 05-27-2009, 10:21 PM   #8
dfresh4130
LQ Newbie
 
Registered: May 2009
Posts: 6

Original Poster
Rep: Reputation: 0
Example of original logs:

Tue May 26 00:00:04 CDT 2009|646|getSubsInfo|jds-JDS:124331400418735
Tue May 26 00:00:05 CDT 2009|472|getSubsInfo|jds-JDS:124331400508540
Tue May 26 00:00:05 CDT 2009|217|authenticate|jds
Tue May 26 00:00:05 CDT 2009|146|getBanInfo|jds-JDS:124331400563943
Tue May 26 00:00:05 CDT 2009|164|getSubsInfo|jds-JDS:124331400569744

After manually editing with vi:

Tue May 26 00:00:04 CDT 2009|646|getSubsInfo|124331400418735
Tue May 26 00:00:05 CDT 2009|472|getSubsInfo|124331400508540
Tue May 26 00:00:05 CDT 2009|217|authenticate
Tue May 26 00:00:05 CDT 2009|146|getBanInfo|124331400563943
Tue May 26 00:00:05 CDT 2009|164|getSubsInfo|124331400569744

After editing with first sed command of: sed 's/jds-JDS://g' file > edited1

Tue May 26 00:00:04 CDT 2009|646|getSubsInfo|124331400418735
Tue May 26 00:00:05 CDT 2009|472|getSubsInfo|124331400508540
Tue May 26 00:00:05 CDT 2009|217|authenticate|jds
Tue May 26 00:00:05 CDT 2009|146|getBanInfo|124331400563943
Tue May 26 00:00:05 CDT 2009|164|getSubsInfo|124331400569744

After editing with second sed command of: sed 's/\|jds//g' edited1 > edited2

Tue May 26 00:00:04 CDT 2009|646|getSubsInfo|124331400418735
Tue May 26 00:00:05 CDT 2009|472|getSubsInfo|124331400508540
Tue May 26 00:00:05 CDT 2009|217|authenticate
Tue May 26 00:00:05 CDT 2009|146|getBanInfo|124331400563943
Tue May 26 00:00:05 CDT 2009|164|getSubsInfo|124331400569744


I've been trying to read up and understand perl, but it seems mostly greek to me. I'm doing well enough with writing and understanding shell scripts. This is a fairly important report so I'm just looking for the quickest solution. I come from a networking background so scripting/programming isn't my normal area of knowledge.
 
Old 05-27-2009, 10:31 PM   #9
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,696
Blog Entries: 5

Rep: Reputation: 241Reputation: 241Reputation: 241
Quote:
Originally Posted by syg00 View Post
That may be true, but I can understand the OP not wanting to screw around with something that worked.
script development should be done in a development environment, tested with testing plans, make sure it is working and then accepted by management before bringing to production.
 
Old 05-27-2009, 10:33 PM   #10
billymayday
Guru
 
Registered: Mar 2006
Location: Sydney, Australia
Distribution: Fedora, CentOS, OpenSuse, Slack, Gentoo, Debian, Arch, PCBSD
Posts: 6,678

Rep: Reputation: 122Reputation: 122
Do you get anything interesting if you try

diff manually_edited_file sed_edited_file

?
 
Old 05-27-2009, 10:38 PM   #11
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,696
Blog Entries: 5

Rep: Reputation: 241Reputation: 241Reputation: 241
make sure you put $.
Code:
sed 's/\|jds$//g'
 
Old 05-27-2009, 10:50 PM   #12
dfresh4130
LQ Newbie
 
Registered: May 2009
Posts: 6

Original Poster
Rep: Reputation: 0
Nope, don't get any differences between the manually and sed edited files. Even just tried using the sed 's/\|jds$//g' command and got the same thing.
 
Old 05-27-2009, 10:59 PM   #13
billymayday
Guru
 
Registered: Mar 2006
Location: Sydney, Australia
Distribution: Fedora, CentOS, OpenSuse, Slack, Gentoo, Debian, Arch, PCBSD
Posts: 6,678

Rep: Reputation: 122Reputation: 122
How about if you edit in place using sed's -i option?
 
Old 05-27-2009, 11:42 PM   #14
dfresh4130
LQ Newbie
 
Registered: May 2009
Posts: 6

Original Poster
Rep: Reputation: 0
The version of sed I have doesn't have a -i option. This is an AIX 5.3 box.

$> what /usr/bin/sed
/usr/bin/sed:
61 1.15 src/bos/usr/ccs/lib/libc/__threads_init.c, libcthrd, bos53H, h2006_10B1 3/5/06 16:44:58
24 1.38 src/bos/usr/bin/sed/sed0.c, cmdedit, bos530 8/27/03 04:21:19
35 1.14.3.1 src/bos/usr/bin/sed/sed1.c, cmdedit, bos53N, n2007_33A6 6/28/07 01:57:03
 
Old 05-28-2009, 12:38 AM   #15
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,289

Rep: Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034Reputation: 2034
You could show us the Perl script, unless its massive.
Anyway, in Perl, normally you'd read each line into a var like $rec and then amend it:
Code:
$rec =~ s/jds-JDS:// ;
$rec =~ s/\|jds// ;
Even showing us the section of code where the file is read in would help.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Replacing text in multiple files ? centosfan Linux - General 10 09-20-2008 07:25 AM
Replacing text on specific lines with sed or awk? Lantzvillian Linux - Newbie 5 10-17-2007 09:00 AM
Messing with sed, need some info on changing text in files MasterC Linux - Software 1 07-27-2007 06:12 PM
sed with large text files? apollyonus Linux - Server 3 03-22-2007 07:33 AM


All times are GMT -5. The time now is 07:21 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration