Translating Standard Scripts to Linux
 

I've been translating my code from Windows XP DOS BATCH files to Linux BASH shell scripts. I've run into a problem from this script. I'm having trouble parsing the database I've created, which tells Linux how to create a set of directories in this case.

The I'm on mknasosnames2.sh, which it's job is to create standard directories for each known operating system. It's code in Windows XP:

It's Linux code so far:

It fails on the second while loop, currently.

By the way, what my standard scripts are is this: Any OS that can support the standard scripts can be used on this network, I know this will work in Linux, I just don't know how yet.

If you're testing for a directory, use -d not -f in your test.

# if [ -d $line2[0] ]; then

This is it's output:

Thanks. I'll use -d instead of -f. But with the output, now you see the problem. I'd tried a for loop, and it does the same thing. Appearently, it's comming from the cat command.

---------- Post added 02-04-13 at 03:38 AM ----------

If it really sees that shortened version of what it reports, it has reason to complain, but I want it to see the whole path, which I put in the file.

This page has some tips about converting DOS batch files to bash.

http://www.tldp.org/LDP/abs/html/dosbatch.html

But if you really want to code "standard" scripts, then you have to restrict all of your code to what's available in the POSIX standard. When you run a script with "#!/bin/sh" at the top, it will be interpreted as such, even if the underlying interpreter is actually bash or another shell.

Conversely, to get the full set of bash features, you have to use #!/bin/bash.


POSIX doesn't have many of the useful modern features that are available to bash, such as arrays. See this page for bash-specific commands and their equivalents.

http://mywiki.wooledge.org/Bashism


Note: If your script does include things like arrays, and they seem to work, that's only because the system is still set to use bash (or a similar shell) to do the interpreting. Since arrays are not counter to POSIX, but just undefined by it, there's nothing keeping a shell from interpreting them if it wants to. But if, for example, the system was using dash instead (a strictly complying shell), it would break. All /bin/sh says is that, if a script is coded according to POSIX, it will run. It doesn't try to force compliance to it.

In fact, one of the best ways to test that your script is POSIX-compliant is to use #!/bin/dash as your shebang. If it runs in dash, it should run anywhere.


Incidentally, I see several errors in the above script. Unquoted variables, improperly formatted array expansions, Useless Use Of Cat. But maybe I should wait for you to clarify whether you actually need POSIX or not, or whether a bash-compatible script would do (it is available standard on pretty much every Linux distro, after all). The latter would certainly be much more convenient.

PS: You also really need to supply us with a sample of the input. We need to see what the files that it's processing hold.

really.. it's hard to say how to fix it if you don't say what's actually wrong.. second loop is wrong it like my pc is broken fix it.. for file read i use
for f in $(cat -s ~/bin/BabyPhotoStarGalleryGetImagesAslist.sh); do echo "LINE".$f;done;
but there is a problem f is each word(1) separated by spaces
so to read lines is better to use readarray f < ~/bin/BabyPhotoStarGalleryGetImagesAslist.sh where f[N] is lines from file

@Sunnydrake

Please Do not Read Lines With For. Always use a while+read loop instead.

How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?
http://mywiki.wooledge.org/BashFAQ/001

readarray/mapfile is another safe option, but probably not suitable if the input file is very large. It's also a bash-specific extension (available since 4.0), and not at all portable.


And do please use ***[code][/code]*** tags around your code and data, to preserve the original formatting and to improve readability.

For some reason it isn't reading line2 correctly when it cats ${line[1]}. Cat you post a bit of /etc/settings/Operating_Systems/dos.sdr ?

I would tend to use something like:

while read -a line2
do
...
done < ${line[1]}

in these situations.

Also, if you use mkdir with the -p (make parent) flag it will surpress the already exists warnings.

can the op please post a few lines of the input file and what they are expecting the output to be ?

I'll post the files.

/etc/settings/Operating_Systems/osnames.sdr

/etc/settings/Operating_Systems/dos.sdr
/etc/settings/Operating_Systems/linux/types.sdr
/etc/settings/Operating_Systems/linux/Mandrake.sdr
/etc/settings/Operating_Systems/linux/Mandriva.sdr
/etc/settings/Operating_Systems/windows/types.sdr
/etc/settings/Operating_Systems/windows/XP.sdr
These are the files I'm trying to parse. I did try it with the following format:

...and this had exactly the same result.

Let me know if I need to explain the syntax of any of these files. Also note that they probably need to be updated to contain my latest information, however, that was not my first priority. It's the same syntax I have on Windows XP, except with a Linux path name instead of a Windows path name. The Windows code works.

You can also see that the code is still not complete yet, when it gets to the second loop working, but that's what I'm stuck on for now.

Interesting thought ...

... has no one in Linux-land ever tackled the ("if I on-ly had a brain...") ;) problem of dealing with DOS batch-files? Is it really true that no one ever wrote a command that could "gracefully and graciously" (of course...) accept DOS batch-files as input, and "have a good college try" at running them?

Casually googling, I didn't find one (yet), but ... it seems so obvious. Really?

You mean, I can't do: #!/bin/ihavenobrain ?? ;)

This works for me.

ls OS
95/ 98/ FreeDOS/ MS-DOS/ Mandrake/ Mandriva/ PC-DOS/ XP/

Note that I've changed all the $line2[0] to ${line2[0]}

It still doesn't work for me for some reason.

Output:

Why would it work there and not here?

It seems to be cutting off the name:

No such file or directoryng_Systems/linux/types.sdr

Which OS is this? Maybe you need to use a newer shell like bash for the script?

That usually indicates carriage returns, try running dos2unix on both the script and input (.sdr) file.

regarding article
$(<afile); is improper use of < :) + !!!FOR IN LOOP!!! problem lies in IN instruction that read array separated by spaces!!!
remember FOR is just cycle construct not some evil overbuggy lib :) you can do stupid mistakes with while loop too :)

most efficient / right path is place for competition.. but i don't want to write in bash sub program to read bytes in cache then determine end-line characters.
PS: thanks for read -r :) my man is a little bit messed up with C header functions.. readarray all that i quick found upto job specs working good. hmm strange it's not part of coreutils .. also lines in bash man page help make doubts about using it on large files.

I ran dos2unix on those files and now it works! Now to finish the rest of the code.

Wine ( http://www.winehq.org/ ) exist for ages, doesn't it ? Also there is DOSEMU: http://www.dosemu.org/ .

1)wineconsole cmd
2)DOSBOX

Just because you can do something one way doesn't mean you should do it that way. I've had this debate before, and it has always been my position (and of most experienced scripters, as represented by the link I gave) that the for loop is the wrong way to go about it.


The main issue I have with it is that proper use of it relies on shell word splitting, which means that the coder must know in advance that the file is in the proper format, and will generate exactly the list of word tokens necessary for the loop. One small mistake there and you have errors.

The second issue is, indeed, with inefficiency and the possibility of hitting the ARG_MAX limit of your system. I consider this a secondary problem, as this unlikely to be a hard limit in most scripts. Still, there is potential for it.

In any case, it's much better in the long run to just do it the right way all the time, and use a proper while+read loop. With practice and experience it's no more difficult to write one of those than a for loop; and it's much safer, flexible, and efficient in all situations. In any case, calling it a "sub program" is a serious misnomer. It's just another type of loop, not any more complex to set up than any other.

Finally, and most importantly here, while you are certainly free to do whatever you want in your own private coding, when you're giving advice in a help forum like this it's your responsibility to always give the best advice you are capable of. That means attempting to show others correct practice, and not your lazy shortcuts (although certainly ignorance of correct practice is forgivable).

That's why I harp on such things so often. I consider it a duty to educate and to weed out poor scripting practice whenever I can. There are generally very good reasons why experienced coders always tend to advise the use of or avoidance of certain structures, and you ignore their hard-earned experience at your peril.


PS: "$( <file )" is simply a bash built-in convenience feature that behaves in exactly the same manner as "$( cat <file )". It only works inside command substitution brackets, and is certainly not portable.

readarray (a.k.a. mapfile, apparently the proper name) is also a bash built-in keyword, as it has to be in order to set environment values directly. So no, you'll never find it in the coreutils. It's only function is to safely load an array with lines from a file, so as with any other variable usage, the limit is in the amount of RAM you have available in which to store the text. As long as you know you can stay inside that limit, it's certainly a viable alternative to a while loop.

hey i just posted mistakes that i found in that article.
< is stdio input redirection which is not cat :) so construct $(< ) is illogical(call/fork shell with just input from file? try using just <filename in shell not like cat result?:)).(and tend to have some hidden caveats(like double input,special chars,etc....) found some when i searched correct instructions for file read).
I can't and I won't do all work for poster... i shared exp and some quick search data that in my tests(yes i wrote mini tests for this specific scenario) made usable expected result).
Each specific task have not so many best approaches and big a number of still useful. I made working suggestion in given(!) task params.
You frame me that it's my lazy shortcuts? you make me smile:) Also don't push on while loop you sound like religion priest, let's not start flame wars. In bare bones this is just same loops with almost identical code(for with read -a ? not a problem :)).
However i can understand you if you say read -a is a good way to read files... despite time out's i mentioned(and yet untested memory usage) it's a challenge to find another std useful file read line tool/command usable in shell... my search ended with readarray (but i tested shells stdio redirection,for in,cat's keys scenarios which not produced useful results.. ).. i hope we not hijacking this thread too much.

What happens if you use a mkdir -p instead of a simple mkdir?

It's not a mistake. As I (and the link) said, "$(<file)" is a minor bash-specific convenience feature. It only works in bash, only works inside command substitution brackets, and it behaves exactly as if there's an invisible cat command in front of it.

Any other attempt at raw redirection (unassociated with a command) will fail, just as you expect.

I believe I explained my reasoning clearly. I will always, to the best of my ability, oppose poor coding practices. And I will continue to correct them any time anyone posts them here. If that marks me as a "religion priest", then so be it.

Whatever you use yourself in private is your own business, and you will accept the consequences of those choices until such time as you learn better. But I will do what I can to ensure that the new, inexperienced scripters who come here to learn do not get taught to use and perpetuate those same errors.

If it's not directly built into the shell, then there's no way you'll ever be able to directly add anything to the shell environment with it. All external commands run as separate processes, which by definition are isolated from the original environment and unable to affect it. Therefore the only options you have are variations of read, redirection, and command substitution.

Sounds like a perfect suggestion for one of those "I just learned <language_of_the_day>. Now what problem can I solve?" questions.
--- rod.

Sorry that took so long, but I needed then to finish coding the rest of it, and it was not possible to entirely test everything, without first finishing a draft of the scripts so that everything is coded.

My focus now turns to the logical test of the scripts, and one script is giving me trouble again. This is a closely related cousin of the script above, but there are some differences.

From here on out, I can assume that the standard scripts will always be stored in /standard_scripts. Here is the code that's giving me trouble now:

mknasosnames:
Output:

In post #12, above, you state that /etc/settings/Operating_Systems/linux/Mandriva.sdr contains
Thus, when processing the 2008.0, line[0]=2008.0 and line[1] is null, so the redirection done < ${line[1]} is somewhat ambiguous, don't you think?

By the way, I notice that your code often uses commands (like, e.g., mkdir) but hardly ever follows the command with a [ $? -ne 0 ] && echo "Could not ..." assert check. You might find the information about the bash built in trap function of interest.

Also, I find your use of the pushd and popd built in functions somewhat inconsistent. Why not use them everywhere you now use a cd command? Or, if you're simply using the pushd command as a placeholder for the current directory, you could just do a variable_name="$(cwd)" and, to return there, a cd "${variable_name}". Of course, you might need the quotes if you directory names sometimes include blanks, etc., but that not much of a hardship.

Thanks for catching that. I'll work on fixing it. That means I may not be able to use the exact same syntax.

I would recommend that instead of using pushd/popd, try redesigning your code flow to run the necessary commands in "()" subshell brackets. When the subshell exits, its environment is lost, and you're automatically back to where you started without having to do anything.

Example:
Save pushd/popd for when you need more complex backtracking through directories (and in all the years I've been scripting, I've never needed them).

And to reiterate a couple of things:

When you use "#!/bin/sh", the script is interpreted as a POSIX-compliant script.

So if you want to use arrays or other bashisms, be sure to make the shebang "#!/bin/bash", and conversely if you need to use "#!/bin/sh" explicitly, then do not use arrays or other bashisms.

(Watch that you type it correctly, too. No spaces.)


And since I don't think I've yet stated it unambiguously:

QUOTE ALL OF YOUR VARIABLE EXPANSIONS. You should never leave the quotes off a parameter expansion unless you explicitly want the resulting string to be word-split by the shell and possible globbing patterns expanded. This is a vitally important concept in scripting, so train yourself to do it correctly now. You can learn about the exceptions later.

http://mywiki.wooledge.org/Arguments
http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes

This generally means that if it begins with a "$", then double-quote it. If it's a string that has spaces or other weird characters in it, also single or double quote it.

The general rule of thumb is also to try to quote the longest string possible. e.g.:

I agree with the above comments by "David the H," but - to muddy the water slightly - I like to at least try to make my comments grammatically correct. viz:Another nice habit is to try to direct error and warning messages to stderr (&2). That way a user of the script can redirect its output to some file or pipe and not have that output stream messed up with errors or warnings.

Unless I'm breaking something, I think I have one more script to go. I'll retest after this, just to make sure though. Unfortunately, I'm stuck again and this time with no errors.

File to translate:
Translated files so far:

csoftwarefolders.sh:
mkosnames.sh
mktypes.sh
mknames.sh
mkversions.sh
mknames2.sh
This time it's just a bunch of strange runtime errors, but I haven't caught it yet. The DOS BATCH file is authoritive on the output this should give.

The new syntax of the deeper files is:

XP.sdr
It is the same with the other Linux files that go that deep. none.sdr, is just a blank file. Also, here is my new file, names.sdr:

When I applied the newer syntax to the files, that's what made the ambiguous message go away on the last script.

Here's the correct output:

Sorry about this, the system is saying my posts are too long. I'm going to have to break it up.

Here's the correct output (p1):

(p2):

(p3):

But the output it's giving me is:

Sorry about the messyness of the tree commands, it's just the way they came out, but it should still be readable...

These particular commands must be run in the same shell. Thanks for the suggestion though.

So all this time I've been putting spaces, and it's incorrect? These scripts are supposed to be run in bash. sh, is a link to bash.

This problem is solved now.

mknasosnames2.sh

The problem with script #1, was I needed to run dos2unix on it.

mknasosnames.sh

The problem with the above was that I needed to change the syntax of the files, so it could be unambigous.

mktypes.sh



---------- Post added 02-19-13 at 04:59 PM ----------

The problem with this last one was that it had an extra loop.

Can I ask what delims=$n was for, or if it does really work as expected? I haven't encountered any syntax form like that. Or perhaps it's a new feature?

dive: I suggest placing strings inside double quotes e.g.