Shell scripting: exclamation marks in strings
 

I'm about to go nuts here. I have a series of log files that contain errors. The script I'm working on will extract an error summary, create a unique list of those errors, and then do some other processing. It works fine in most cases. It barfs all over itself if more than one exclamation mark appears (in succession) within the error summary. And this is why:

Step 1) Extract one line error summary from each log file
Step 2) Place each summary line into a file
Step 3) sort the file (with -u)
Step 4) Grep the original logfiles to get a list of the log files affected and count the number of instances of that error

In the fourth step, the wheels come off because the shell wants to expand the exclamation marks (since '!!' = last command). The grep statement is something like this:

grep "${error_summary}" ${current_log_file}

So, I cannot avoid the substitution by using single quotes; that would mean I'd be trying to match ${error_summary} literally.

I'm at a loss. I have to find some way to handle those exclamation marks internally, maintaining them as exclamation marks so I can compare them against the originals with the grep. Am I making sense? Can anyone help?

how about striping the exclamation marks off of both sets so when you compare them there shouldn't be any exlamation marks in the comparison.

That would work, but I'm hesitant to modify the originals. The log files are simulation results, and other individuals might be processing them with their own scripts. They are sort of community property in that way. Chances are good someone will not look favorably at changing the originals, no matter how minor.

Good point as I was thinking internally but since you are using grep to process the file you will have to modify the file. There is always creating a temporary copy of the file that you can massacre. You could always a program that will handle the checking instead of grep in which case you can do what ever you wish.

Not quite sure this helps, but anyways...

But you could change the variable error_summary, replacing all '!' with '\!', so they will escaped when running the grep command, so they will not be touched by the shell.
If this doesn't help, maybe it helps not to change !! into \!\! (as the above code does), but to change it into: \!!

Thanks for the responses.

I like the sed use, but for some reason, I get this error:

/: Event not found.

Here's the code I included:

The idea being that I can change my summary file to match what grep needs to match against the original. Those are the only changes, so the interpreter doesn't like something. I just can't see the problem.

Does sed give you this error?
It tried to reproduce the error, using some random files (as a substitute for your files), like:..but none gives me an error...

Maybe you use some different sed? Or does the error come from other commands in your script?

May be you can try escaping the '!' in the sed-script itself with two extra backslashes (on my system the output is the same:

Yeah, I'm confident the error is from the sed command. I added a series of echo statements so I could see the point of failure. Execution dies when it reaches sed. When I execute the command from a prompt (with filenames specified), I get good results.

That said, I didn't give you guys all the information possible. This is a tcsh script. Even so, I would imagine invoking sed from tcsh should be identical to invoking it from bash.

As another experiment, I tried replacing the forward slashes with %'s like so:

sed 's%!%\\!%g' ${error_summary_file} > ${error_summary_file}.new

When executing that command, the scripts spits out:

0: Event not found.

Perhaps I'll try another delimiter later. I'll try your suggestion about escaping the exclamations within the sed expression and see how that goes.

The system does have bash and I'm considering porting it over if it will solve this problem, but I have yet to try it under bash to know one way or the other.

I've never used (t)csh, but my wild guess would be that this could be a tcsh issue. It seems people generally agree that (t)csh for scripting is not a good idea. I have not read about this it myself, so I have no idea whatsoever if has anything to do with your case, but you may want to read this doc anyway:

Csh Programming Considered Harmful

Maybe you could try porting the troubling part of your script to sh / bash, just to see if it makes a difference.

Or could it be just a quoting issue with tcsh?? Like, the single quote's around the sed script, do they work as one might expect? (I mean if you are used to the quoting semantics of other shells)

A few of the things the article mentioned seemed as though they may have roots in the same thing I'm seeing. I also have to admit I have had fewer problems with bash scripting than tcsh, but my experience as a whole is limited. Like the guy pointed out, I was lured by the C-style comparisons, but I'm not sure that tcsh is ready for the scrap heap yet. I'm sure there are issues with bash too. Maybe not as many, but everything has faults/drawbacks.

I probably will port it over to bash. I'll give it a shot tomorrow, and post the results. It just seems odd to me that a shell designed to resemble another strong programming language would suffer from such inconsistencies. But I've seen stranger things.