I've run into an issue here. I have a perl script that takes some logs on a daily basis and creates a report from them. Recently the format of the logs was changed and that's causing the report to give wrong data. I've been trying to have the log files edited to remove the new data causing problems and I've been successful with the sed command. However, whenever I use the perl script on those new logs it gives me out of memory errors. Absolutely no idea why because when I manually edit them it works fine. So whenever I've used sed to touch those logs the perl script blows up. Below is a snippet of the log formats. I need to get rid of the jds-JDS: and jds values on each line. Does anyone know how to do this with another command? I tried cut and tr, but neither of them work the way I needed it to.

Tue May 26 00:00:00 CDT 2009|352|getDetailedUserProfileByUsername|jds
Tue May 26 00:00:02 CDT 2009|896|querySubscriberUnbilledUsageSummaryBtRequestBT|jds
Tue May 26 00:00:02 CDT 2009|1139|querySubscriberUnbilledUsageSummaryBtRequestBT|jds
Tue May 26 00:00:02 CDT 2009|353|getSubsInfo|jds-JDS:124331400223813
Tue May 26 00:00:02 CDT 2009|753|getSubsInfo|jds-JDS:124331400200410
Tue May 26 00:00:03 CDT 2009|1208|getCustSecurityInfo|jds-JDS:124331400201512
Tue May 26 00:00:03 CDT 2009|1301|getBanInfo|jds-JDS:124331400201411
Tue May 26 00:00:03 CDT 2009|734|getSubsInfo|jds-JDS:124331400297914
Tue May 26 00:00:03 CDT 2009|219|authenticate|jds
Tue May 26 00:00:03 CDT 2009|294|getDetailedUserProfileByUsername|jds
Tue May 26 00:00:04 CDT 2009|207|getSubsInfo|jds-JDS:124331400390717
Tue May 26 00:00:04 CDT 2009|460|getSubsInfo|jds-JDS:124331400378816
Tue May 26 00:00:05 CDT 2009|494|getSubsInfo|jds-JDS:124331400512718
Tue May 26 00:00:05 CDT 2009|196|getBanInfo|jds-JDS:124331400567820
Tue May 26 00:00:05 CDT 2009|426|getSubsInfo|jds-JDS:124331400553219
Tue May 26 00:00:06 CDT 2009|2267|getBanInfo|jds-JDS:124331400374615
Tue May 26 00:00:06 CDT 2009|519|getBanInfo|jds-JDS:124331400573622
Tue May 26 00:00:06 CDT 2009|556|getSubsInfo|jds-JDS:124331400573621
Tue May 26 00:00:06 CDT 2009|219|getBanInfo|jds-JDS:124331400608724
Tue May 26 00:00:06 CDT 2009|685|getCustSecurityInfo|jds-JDS:124331400573623
Tue May 26 00:00:06 CDT 2009|267|getBanInfo|jds-JDS:124331400622726
Tue May 26 00:00:06 CDT 2009|566|getCustSecurityInfo|jds-JDS:124331400622827

Why don't you just fix the perl script to accept the new input format?

Or better, make it insensitive to format changes. You could also do the mangling in that perl script prior to the current processing - anything you can do in sed perl will handle.

If I could fix the perl script I would. I'm no perl expert and the script is a remnant left over from a previous coworker who was a perl guru and his programming skills were cryptic to say the least.

Could you show a line before and after manual edit, before and after sed edit and the sed command you use?

don't really need expert to do replacement of strings with Perl. Since you are paid (is it not??) to do your job, its only natural you start reading up on Perl because eventually you are going to maintain those Perl scripts your predecessor has written.

That may be true, but I can understand the OP not wanting to screw around with something that worked.
Mangling the input should work - I'm also interested to what could cause this from sed usage.

Example of original logs:

Tue May 26 00:00:04 CDT 2009|646|getSubsInfo|jds-JDS:124331400418735
Tue May 26 00:00:05 CDT 2009|472|getSubsInfo|jds-JDS:124331400508540
Tue May 26 00:00:05 CDT 2009|217|authenticate|jds
Tue May 26 00:00:05 CDT 2009|146|getBanInfo|jds-JDS:124331400563943
Tue May 26 00:00:05 CDT 2009|164|getSubsInfo|jds-JDS:124331400569744

After manually editing with vi:

Tue May 26 00:00:04 CDT 2009|646|getSubsInfo|124331400418735
Tue May 26 00:00:05 CDT 2009|472|getSubsInfo|124331400508540
Tue May 26 00:00:05 CDT 2009|217|authenticate
Tue May 26 00:00:05 CDT 2009|146|getBanInfo|124331400563943
Tue May 26 00:00:05 CDT 2009|164|getSubsInfo|124331400569744

After editing with first sed command of: sed 's/jds-JDS://g' file > edited1

Tue May 26 00:00:04 CDT 2009|646|getSubsInfo|124331400418735
Tue May 26 00:00:05 CDT 2009|472|getSubsInfo|124331400508540
Tue May 26 00:00:05 CDT 2009|217|authenticate|jds
Tue May 26 00:00:05 CDT 2009|146|getBanInfo|124331400563943
Tue May 26 00:00:05 CDT 2009|164|getSubsInfo|124331400569744

After editing with second sed command of: sed 's/\|jds//g' edited1 > edited2

Tue May 26 00:00:04 CDT 2009|646|getSubsInfo|124331400418735
Tue May 26 00:00:05 CDT 2009|472|getSubsInfo|124331400508540
Tue May 26 00:00:05 CDT 2009|217|authenticate
Tue May 26 00:00:05 CDT 2009|146|getBanInfo|124331400563943
Tue May 26 00:00:05 CDT 2009|164|getSubsInfo|124331400569744


I've been trying to read up and understand perl, but it seems mostly greek to me. I'm doing well enough with writing and understanding shell scripts. This is a fairly important report so I'm just looking for the quickest solution. I come from a networking background so scripting/programming isn't my normal area of knowledge.

script development should be done in a development environment, tested with testing plans, make sure it is working and then accepted by management before bringing to production.

Do you get anything interesting if you try

diff manually_edited_file sed_edited_file

?

make sure you put $.

Nope, don't get any differences between the manually and sed edited files. Even just tried using the sed 's/\|jds$//g' command and got the same thing.

How about if you edit in place using sed's -i option?

The version of sed I have doesn't have a -i option. This is an AIX 5.3 box.

$> what /usr/bin/sed
/usr/bin/sed:
61 1.15 src/bos/usr/ccs/lib/libc/__threads_init.c, libcthrd, bos53H, h2006_10B1 3/5/06 16:44:58
24 1.38 src/bos/usr/bin/sed/sed0.c, cmdedit, bos530 8/27/03 04:21:19
35 1.14.3.1 src/bos/usr/bin/sed/sed1.c, cmdedit, bos53N, n2007_33A6 6/28/07 01:57:03

You could show us the Perl script, unless its massive.
Anyway, in Perl, normally you'd read each line into a var like $rec and then amend it:
Even showing us the section of code where the file is read in would help.