Bash redirection being taken literally by programs

R00ts · 05-24-2005, 01:17 PM

I'm a little stumped over the behavior of one of my bash scripts. I call a series of programs in this script and on the lines I execute those programs I give the input/output redirection ('<' '>' or '2>'), but the programs are interpretting this literally as arguments to the program and are exhibiting incorrect behavior, such as:

1) Printing a message that the argument '<' was not recognized
2) Printing to stdout instead of redirecting
3) Generating seg faults because it did not recieve it's expected input

I echo out the exact line everytime I call the program and it's absolutely correct. Furthermore, when I copy this output line on the command line the redirection works and the program happily runs. Does anyone have a clue as to what may be causing this?

The only thing I can think of is maybe because this script I'm running calls another script multiple times which in turns calls several other programs and scripts, and somewhere somehow the re-direction is getting messed up between the pass calls. (That's 3 levels in the script hierarchy if you weren't counting). (I literally pass "/home/me/bin/prog < input.i > output.o 2> error.e" from the top-level script down to the mid-level, and the mid-level passes it down to the bottom-level program/script.

Thanks in advance for any clues you can give as to what may be going wrong. I've been working on this problem since yesterday evening and I still haven't been able to resolve the issue.

jlliagre · 05-24-2005, 01:23 PM

This is normal behaviour, if your target programs get the redirection symbols as parameters, they almost always can't do anything with them.

The problem is certainly due to your script quoting the redirections, thus hiding them to the shell, which is the one that should have handled them.

R00ts · 05-24-2005, 01:36 PM

Yeah, after thinking about it that sounds right. It's redirecting the output of the mid-level script instead of the low-level program that the mid-level script calls. Hmmm, is there a way to safely pass redirection arguments down through a script? I pass them as a simple quoted string when I send it from the high-level to mid-level script.

R00ts · 05-24-2005, 01:48 PM

One fix I thought of would be to create an option in my mid-level script that grabs the name of the file for input, output, and error like so:

Code:

./midscript -i input.i -o output.o -e error.e

And then when I call my bottom level scripts/programs I can give them the redirection plus the file name. However this seems inefficient and would require a significant amount of work for me to re-structure my top level script (which contains about 50 unique calls to the mid-level script) and I'd have to add this functionality in the mid-level script.

Does anyone know a way that I can tell my mid-level script "Ok, you got these redirection arguments but don't interpret them, because they aren't for you."? I hope so, but a part of me doubts it. Maybe using a `` or '' string instead of a "" string when I pass the program + redirections from top level to mid-level would do it?

jlliagre · 05-24-2005, 01:56 PM

Can you post a small sample of your scripts, I mean the smaller and simpler you can write that still exhibits the issue ?

R00ts · 05-24-2005, 02:19 PM

My scripts are really huge and intricate (hunderds of line each). But I think I found the solution to the problem from someone at work. Basically you put a \ before the re-direction arguments for each level of redirection and that prevents the script from using the re-direction. I'm not sure if I'm implementing the way he told me to, but I wrote a sample script and it seems like it works for my test case. I'm going to try putting it in the main script hiearchy right now and if it's successful, I'll post the solution here.

R00ts · 05-24-2005, 05:39 PM

Nope, I couldn't get it to work. I first tried putting one \ in front of each < and > character in passing the script and then using sed in the mid-level script to remove each \ in the argument passed. The command was echoed correctly, but didn't run correctly (same exact problems as before). Then I tried two \\ in front of the < and > and removing them in mid-level, but that didn't work either. Then I tried actually replacing < and > with ? and \ respesctively so they wouldn't get picked up at all and used sed to replace them with < and > in the mid-level script, and again the same problems.

I don't understand why this isn't working. Even in my sample script when I did this the lowest level didn't printed to stdout instead of the output file specified by the redirection. If anyone knows what's going on here I am in dire need of your help.

jlliagre · 05-24-2005, 06:12 PM

post a test case so we could go further.

eddiebaby1023 · 05-25-2005, 07:39 AM

If you start escaping the redirection operators he shell wil strip the '\'s off at each level. Echoing commands may also cause the shell do to some escape-stripping so can't be relied on for veracity. As others have requested, post some examples so we can give you pertinent advice (or at least explain why what you're doing isn't working).

R00ts · 05-25-2005, 10:46 AM

Here's a little demo I whipped up that is a gross simplification of my problem. I call a toplevel script, which calls a mid-level script with redirection arguments not intended for it, which calls a hello world c program that prints to stdout and stderr. Here's the code:

toplevel

Code:

#!/bin/bash

echo "TOP"
./midlevel "\> output.o \2> error.e"
exit 0

midlevel

Code:

#!/bin/bash

echo "MIDDLE"
./hello_world $1
exit 0

hello_world (bottom level program)

Code:

#include <stdio.h>

int main() {
  fprintf(stdout, "Hello, world!\n");
  fprintf(stderr, "Hello, error!\n");
  return 0;
}

So I try to put two \ in front of the redirection args so that the redirection is "reserved" for the hello_world program. Here's the output:

Code:

$ ./toplevel
TOP
MIDDLE
Hello, world!
Hello, error!

Which is obviously incorrect, because "Hello, world!" should go in output.o and "Hello, error!" should go in error.e (neither of these two files were created by running the script). The redirection args are being completely ignored. tried changing the number of backslashes in toplevel to zero, two, three, and four and still got the same result. I'll keep playing with this but I'm running out of ideas on how to make it work.

Thanks for your help guys

Ack, just before I hit reply I figured out the problem. I'm passing the redirection args in quotes "" because the way my mid-level script works is it takes the last argument (which is in quotes) as a string containing the program path + binary, it's arguments, and it's redirection arguments. When I take out the quotes the following are printed to output.o and error.e with no \ in front of the operators:

output.o

Code:

MIDDLE
Hello, world!

error.e

Code:

Hello, error!

Using backslashes makes things worse. "Hello, error!" is never printed to error.e (always stdout) and in one case I actually got MIDDLE and "Hello, world!" printed to error.e.

R00ts · 05-25-2005, 01:14 PM

Ok, I have plan. It's now obvious that whenever redirection is enclosed in quotes it doesn't get operated on. My mid-level script needs quotes in it's last argument for work and it would be a pain in the ass to have to change that. So, what I think I'll do is in the mid-level script I'll use sed to extract the input, output, and error redirection names (if they exist) and then extract them from the quote. Then, when I make the call to the program, I'll see what redirection it needs and put that in by hand. It may be a little inefficient way of doing things, but I can't be delayed by this problem any longer. I'll post back if it works.

jlliagre · 05-25-2005, 03:06 PM

Try that:

Code:

echo "TOP"
./midlevel "> output.o 2> error.e"
exit 0

Code:

echo "MIDDLE"
eval ./helloworld $1
exit 0

R00ts · 05-25-2005, 03:14 PM

You sir, are my hero!!! Everything works great now, and it's just a single word solution! I may have to write a song singing your praise and erecting a few statues in your honor for this.

jlliagre · 05-26-2005, 12:46 AM

Thanks, looking forward for the song ...