[SOLVED] Need some help with a Bash script

LUB997 · 05-03-2010, 05:07 AM

I am having a little trouble with a Bash shell script that I am working on. It is intended to read the information out of a gedcom genealogy file and determine who has lived in a particular area and output a report based upon who lived in a given area (like the different sections of a bygdebok for example). I am having a number of issues.

1) I can't seem to figure out how to get a variable to work as the index of an array. For example, I am trying to use the variable $individual to count which individual is being read in from the .ged file, and once that is figured out, next I will read in the individual's name, so I would like to create an array of names using the variable

Code:

$individual

as the index of the array. However, if I try the code

Code:

name[$individual]=$lineoftext;

where

Code:

$lineoftext

is the name that was read in from the file, the shell doesn't like it and throws the error message at me,

Quote:

")syntax error: invalid arithmetic operator (error token is "

.

2) If I try to echo a statement with multiple variables, the first variable is getting ignored, and I can't figure out why. For example, if my line of code says something like

Code:

echo $individual" "$lineoftext

or

Code:

echo "$individual $lineoftext"

the contents of

Code:

$individual

are not being displayed, but the intended space and contents of

Code:

$lineoftext

are being displayed. I know it's not a problem of

Code:

$individual

not loading its data properly and being empty, because if I put the code

Code:

echo $individual

the contents of the variable

Code:

$individual

display perfectly. Not sure what I'm doing wrong.

Here is an example of the file format I'm dealing with:

Code:

0 @I1039@ INDI
1 NAME Erik Eriksson /Skjeldal/
1 SEX M
1 BIRT
2 DATE 15 FEB 1815
2 PLAC Skjeldal, Stødle, Etne, Hordaland, Norway
1 DEAT
2 DATE 26 JUL 1902
2 PLAC Cambridge, Story, Iowa
1 CENS
2 DATE 1860
2 PLAC Palestine, Story, Iowa
1 CENS
2 DATE 1850
2 PLAC Saratoga, Grundy, Illinois
1 CENS
2 DATE 1856
2 PLAC Union, Story, Iowa
1 CENS
2 DATE 1870
2 PLAC Palestine, Story, Iowa
1 CENS
2 DATE 1880
2 PLAC Palestine, Story, Iowa
1 CENS
2 DATE 1885
2 PLAC Palestine, Story, Iowa
1 CENS
2 DATE 1895
2 PLAC Palestine, Story, Iowa
1 CENS
2 DATE 1900
2 PLAC Union, Story, Iowa
1 CHR
2 DATE 22 FEB 1815
2 PLAC Etne, Hordaland, Norway
1 IMMI
2 DATE 1847
2 PLAC Norway to America
1 EVEN
2 TYPE Ship
2 PLAC Kong Sverre
1 EVEN
2 TYPE Obituary
2 PLAC The Roland Record
2 SOUR Erik Sheldahl En af de ældste Setlere Død
3 CONT  
3 CONT Erik Sheldahl, Cambridge, Ia., asgik ved Døden den 26de Juli, 87 Aar 4 Maaneder og 11 Dage gammel. Han blev begraven Mandag fra Palestina Kirke, hvor Pastor Heimarck forrettede. I Sørgehuset talte Rev
3 CONC . Grave. Der var en stor Menneskemasse tilstede i Kirken, ogsaa fleve langveis fra; thi den Afdøde havde en talrig Slægt og Kjendte, og var en særdeles agtet og afholdt Mand. Han var altid taget virks
3 CONC om Del talt til at bygge op iblandt os, og da i færdeleshed i det Kirkelige, Ved hans Død har et langt og daadrigt Liv fundet en Alslutning.
3 CONT  
3 CONT Erik Sheldahl var født i Etne, Norge, den 15de Februar, 1815. Han reiste til Amerika i 1847 og bosatte sig i Kendall Co., Ill. Her levede han i syv Aar og flyttede saa til Story Co., Iowa, hvor han si
3 CONC den har boet. Han var først gift med Margaret Skjold. I dette Ægteskab blev 8 Børn fødte ham, af hvem 4 lever. Denne hans første Hustru døde i Juni 1859. Han blev igjen gift med Alice Berøen. Dette Æg
3 CONC teskab var velsignet med 12 Børn hvoraf 7 lever. Med Undtagelse af 3, saa lever alle hans Børn i Story County. De tre ere Erik, som lever i Des Moines, Louis i Colorado og Margaret i Syd Dakota.
1 FAMS @F511@
1 FAMS @F513@
1 FAMC @F499@

I only included what the file format looks like for one individual in the file to give a good idea how it is set up, but not to take up too much space on the forum.

Here is the code I have so far in my shell script:

Code:

#!/bin/bash
# Shell script to generate bygdebok sections by location from gedcom files formatted in UTF-8.

inputfile=test.ged
category=0;

while read lineoftext
do

  if [[ ${lineoftext:0:1}    == "0" ]]
  then
    category=0;

  else
    if [[ ${lineoftext:0:1}    == "1" ]]
    then
      category=1;

    else
      if [[ ${lineoftext:0:1}    == "2" ]]
      then
        category=2;
      fi
    fi
  fi

  if [ $category == 0 ]
  then
    if [[ $lineoftext == *\@\ INDI* ]]
    then
      lineoftext=`echo $lineoftext | sed "s/0\ \@I// g"`;
      lineoftext=`echo $lineoftext | sed "s/\@\ INDI// g"`;
      individual=$lineoftext;

    else
      if [[ $lineoftext == *\@\ FAM* ]]
      then
        lineoftext=`echo $lineoftext | sed "s/0\ \@F// g"`;
        lineoftext=`echo $lineoftext | sed "s/\@\ FAM// g"`;
        family=$lineoftext;
      fi
    fi
  fi

  if [ $category == 1 ]
  then
    if [[ $lineoftext == *NAME* ]]
    then
      lineoftext=`echo $lineoftext | sed "s/1\ NAME\ // g"`;
      lineoftext=`echo $lineoftext | sed "s/\/// g"`;
      echo $individual;
      echo $lineoftext;
    fi
  fi

done < $inputfile

exit 0

Here is what my shellscript's output currently looks like which demonstrates that it is reading in and storing the appropriate information in the variables $individual and $lineoftext:

Code:

./generate.sh

Code:

Cecilia Gudmundsdotter Leervig
1922
Jertrud Johannesdotter Bauge
1923
Helen Nessa
1924
Mervin Tesdall
1925
Arnold Tesdall
1926
Herbert Tesdall
1927
Vilma Tesdall
1928
Thomas Nessa
1929
Helen
1930
Thurman Johnson
1931
Frances
1932
Oliver Johnson
1933
Margaret Opal Fritz
1934
Ole Gunnarsson Nernes
1935
Marta Knutsdotter Flåte
1936
Gunnar Olsson Nernes
1937
Siri Torsdotter
1938
Knut Tørrisson Flåte
1939
Rannveig Tjerandsdotter Vågen

I included only the last few lines of output as to not take up too much space because the output goes on forever, but you get the point from the example. This output has nothing to do with what the final output of the program is intended to be like. It is merely a test to prove that the 2 variables in question are getting their date loaded in properly.

Any suggestions appreciated.

grail · 05-03-2010, 05:31 AM

Code:

  if [[ ${lineoftext:0:1}    == "0" ]]
  then
    category=0;

  else
    if [[ ${lineoftext:0:1}    == "1" ]]
    then
      category=1;

    else
      if [[ ${lineoftext:0:1}    == "2" ]]
      then
        category=2;
      fi
    fi
  fi

or

Code:

category=${lineoftext:0:1}

Code:

if [[ $lineoftext == *\@\ INDI* ]]

This may not be wrong but I think it would be clearer to say it contains,ie:

Code:

if [[ $lineoftext =~ *\@\ INDI* ]]

And according to your script, as shown, the only line to deliver output is:

Quote:

echo $individual;
echo $lineoftext;

LUB997 · 05-03-2010, 06:03 AM

Quote:

And according to your script, as shown, the only line to deliver output is:
Quote:
echo $individual;
echo $lineoftext;

Right, but what I was trying to say, maybe I wasn't as clear as I intended, is that those two lines are only there to test whether or not the variables are loading their data correctly. They are only temporary and have no other purpose. If I remove them completely since they have no purpose other than testing and the code is changed to say:

Code:

# Shell script to generate bygdebok sections by location from gedcom files formatted in UTF-8.

inputfile=test.ged
category=0;

while read lineoftext
do

  if [[ ${lineoftext:0:1}    == "0" ]]
  then
    category=0;

  else
    if [[ ${lineoftext:0:1}    == "1" ]]
    then
      category=1;

    else
      if [[ ${lineoftext:0:1}    == "2" ]]
      then
        category=2;
      fi
    fi
  fi

  if [ $category == 0 ]
  then
    if [[ $lineoftext == *\@\ INDI* ]]
    then
      lineoftext=`echo $lineoftext | sed "s/0\ \@I// g"`;
      lineoftext=`echo $lineoftext | sed "s/\@\ INDI// g"`;
      individual=$lineoftext;

    else
      if [[ $lineoftext == *\@\ FAM* ]]
      then
        lineoftext=`echo $lineoftext | sed "s/0\ \@F// g"`;
        lineoftext=`echo $lineoftext | sed "s/\@\ FAM// g"`;
        family=$lineoftext;
      fi
    fi
  fi

  if [ $category == 1 ]
  then
    if [[ $lineoftext == *NAME* ]]
    then
      lineoftext=`echo $lineoftext | sed "s/1\ NAME\ // g"`;
      lineoftext=`echo $lineoftext | sed "s/\/// g"`;
      name[$individual]=$lineoftext;
    fi
  fi

done < $inputfile

exit 0

The result is:

Quote:

")syntax error: invalid arithmetic operator (error token is "

So, the first thing I need help on is how to use my variable $individual (stores the number of person being read in) as the index in the name array, as the code

Code:

name[$individual]=

is not working.

Also, if I change the code to say:

Code:

#!/bin/bash
# Shell script to generate bygdebok sections by location from gedcom files formatted in UTF-8.

inputfile=test.ged
category=0;

while read lineoftext
do

  if [[ ${lineoftext:0:1}    == "0" ]]
  then
    category=0;

  else
    if [[ ${lineoftext:0:1}    == "1" ]]
    then
      category=1;

    else
      if [[ ${lineoftext:0:1}    == "2" ]]
      then
        category=2;
      fi
    fi
  fi

  if [ $category == 0 ]
  then
    if [[ $lineoftext == *\@\ INDI* ]]
    then
      lineoftext=`echo $lineoftext | sed "s/0\ \@I// g"`;
      lineoftext=`echo $lineoftext | sed "s/\@\ INDI// g"`;
      individual=$lineoftext;

    else
      if [[ $lineoftext == *\@\ FAM* ]]
      then
        lineoftext=`echo $lineoftext | sed "s/0\ \@F// g"`;
        lineoftext=`echo $lineoftext | sed "s/\@\ FAM// g"`;
        family=$lineoftext;
      fi
    fi
  fi

  if [ $category == 1 ]
  then
    if [[ $lineoftext == *NAME* ]]
    then
      lineoftext=`echo $lineoftext | sed "s/1\ NAME\ // g"`;
      lineoftext=`echo $lineoftext | sed "s/\/// g"`;
      echo $individual" "$lineoftext;
    fi
  fi

done < $inputfile

exit 0

or

Code:

#!/bin/bash
# Shell script to generate bygdebok sections by location from gedcom files formatted in UTF-8.

inputfile=test.ged
category=0;

while read lineoftext
do

  if [[ ${lineoftext:0:1}    == "0" ]]
  then
    category=0;

  else
    if [[ ${lineoftext:0:1}    == "1" ]]
    then
      category=1;

    else
      if [[ ${lineoftext:0:1}    == "2" ]]
      then
        category=2;
      fi
    fi
  fi

  if [ $category == 0 ]
  then
    if [[ $lineoftext == *\@\ INDI* ]]
    then
      lineoftext=`echo $lineoftext | sed "s/0\ \@I// g"`;
      lineoftext=`echo $lineoftext | sed "s/\@\ INDI// g"`;
      individual=$lineoftext;

    else
      if [[ $lineoftext == *\@\ FAM* ]]
      then
        lineoftext=`echo $lineoftext | sed "s/0\ \@F// g"`;
        lineoftext=`echo $lineoftext | sed "s/\@\ FAM// g"`;
        family=$lineoftext;
      fi
    fi
  fi

  if [ $category == 1 ]
  then
    if [[ $lineoftext == *NAME* ]]
    then
      lineoftext=`echo $lineoftext | sed "s/1\ NAME\ // g"`;
      lineoftext=`echo $lineoftext | sed "s/\/// g"`;
      echo "$individual $lineoftext";
    fi
  fi

done < $inputfile

exit 0

the output for each individual is looking like:

Code:

 Åmund Torsteinsson Osvåg

when it should be looking like:

Code:

1039 Åmund Torsteinsson Osvåg

Or in other words, the first variable, $individual, is getting ignored completely in the output of the echo command, and I need help getting it to not be ignored.

grail · 05-03-2010, 06:36 AM

Right so you do know that bash arrays can only have numeric indices?

smoker · 05-03-2010, 06:38 AM

You are not defining $individual inside the last if statement, and you can't use a name as an index in an array, only numbers. Maybe if you could create a hash then it might work.

LUB997 · 05-03-2010, 02:00 PM

Quote:

Right so you do know that bash arrays can only have numeric indices?

Yes, and unless I am misunderstanding something (which is possible), the content of $individual should be numeric as it is set to zero at the beginning and is incremented by one every time a line starting with 0 (a new individual) is read in. Are you implying that perhaps the contents of the variable is not stored as a number but as text and I need to do something to turn it into a number? If so, what should I do to accomplish that? I'm experienced with C#, but not with shell scripts, so the more explanation the better; I have read a lot about Bash variables, but still don't have a great understanding of them evidently.

Quote:

You are not defining $individual inside the last if statement, and you can't use a name as an index in an array, only numbers. Maybe if you could create a hash then it might work.

That's because I don't want $individual to be defined within the last if statement. I want it to be defined when a line beginning with zero and containing the text @ INDI is encountered. The last if statement is for the situation in which a line begins in 1, not 0. A name will never be read from a line beginning with 0, only lines beginning with 1 due to the heirarchy of how the file format is set up. Is this a problem of the variable being only a local variable because of the if statement that it is initiated in? If so, how do I fix that in a shell script and make it a variable that can be accessed anywhere in the script? Is there perhaps a better way to go about it? What is a hash? Can you explain to me how to use a hash in this situation and what it would do?

smoker · 05-03-2010, 02:39 PM

From your first post

Code:

  if [ $category == 1 ]
  then
    if [[ $lineoftext == *NAME* ]]
    then
      lineoftext=`echo $lineoftext | sed "s/1\ NAME\ // g"`;
      lineoftext=`echo $lineoftext | sed "s/\/// g"`;
      echo $individual;
      echo $lineoftext;
    fi
  fi

done < $inputfile

exit 0

Which gave output like

Code:

Cecilia Gudmundsdotter Leervig
1922
Jertrud Johannesdotter Bauge
1923
Helen Nessa
1924
Mervin Tesdall
1925
Arnold Tesdall
1926

That doesn't look like numbers to me. In fact you said what it was supposed to be

Quote:

Here is what my shellscript's output currently looks like which demonstrates that it is reading in and storing the appropriate information in the variables $individual and $lineoftext:

You can't use hashes directly in bash, you have to fake them.

A hash is a construction like an array, but instead of having numeric indices, you have named keys.

So to access a value from a hash you might request the value stored in the hash name -> key "name"

have a look at http://tldp.org/LDP/abs/html/

(use Ctrl + F and search for hash)

LUB997 · 05-03-2010, 03:14 PM

I looked over the information you gave me about hash, and I think that looks like a much better way to go about it and I think it will solve the array problem once I learn how to use hash. Thanks for that information.

Still having problems outputting the number stored in $individual. You expressed doubt that a number is being stored in $individual. If we use the following code:

Code:

#!/bin/bash
# Shell script to generate bygdebok sections by location from gedcom files formatted in UTF-8.

inputfile=test.ged;
category=0;
individual=0;
family=0;

while read lineoftext
do

  if [[ ${lineoftext:0:1}    == "0" ]]
  then
    category=0;

  else
    if [[ ${lineoftext:0:1}    == "1" ]]
    then
      category=1;

    else
      if [[ ${lineoftext:0:1}    == "2" ]]
      then
        category=2;
      fi
    fi
  fi

  if [ $category == 0 ]
  then
    if [[ $lineoftext == *\@\ INDI* ]]
    then
      lineoftext=`echo $lineoftext | sed "s/0\ \@I// g"`;
      lineoftext=`echo $lineoftext | sed "s/\@\ INDI// g"`;
      individual=$lineoftext;

    else
      if [[ $lineoftext == *\@\ FAM* ]]
      then
        lineoftext=`echo $lineoftext | sed "s/0\ \@F// g"`;
        lineoftext=`echo $lineoftext | sed "s/\@\ FAM// g"`;
        family=$lineoftext;
      fi
    fi
  fi

  if [ $category == 1 ]
  then
    if [[ $lineoftext == *NAME* ]]
    then
      lineoftext=`echo $lineoftext | sed "s/1\ NAME\ // g"`;
      lineoftext=`echo $lineoftext | sed "s/\/// g"`;
      echo $individual;
    fi
  fi

done < $inputfile

exit 0

the output looks like:

Code:

0001
0002
0003
0004
0005
0006
0007
0008
0009
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
etc... and on forever

Those are clearly numbers. Not sure why I can echo the value of $individual perfectly fine by itself, but not combined with another variable in an echo statement or it gets ignored.

David the H. · 05-03-2010, 03:43 PM

Bash v4 has associative arrays, which means you can use strings as index elements.

To tell the truth, I'm having a bit of a hard time following this thread. I don't have the energy currently to work through all the code.

But the initial array stuff looks ok to me. It looks like the main problem is getting the variables set properly. Make sure you're watching your quoting.

One thing that might help is to start by declaring the "$individual" variable as an integer before using it. You can also try initiating the array name as well. I believe this is necessary for associative arrays.

I ran a quick test with the following, which simply outputs a list of $RANDOM numbers. Perhaps it will help.

Code:

#!/bin/bash


declare -a array    #declare as a regular index array
declare -i index    #declare as an integer-only variable
declare string      


for index in {1..10}; do         #loop through a list of 10 integers

     string="$RANDOM"
     array[$index]="$string"     #Set each array element to a random number

done

for index in ${!array[@]}; do     #${!array[@]} gives you a list of all array indexes

     echo "The random number is : ${array[$index]}"      #echo all array elements

done

Or this one using bash 4's associative arrays.

Code:

declare -A array     #declare as an associative array
declare index        #declare as a regular variable
declare string


for index in Fry Leela Bender ; do

     string="Futurama character"
     array[$index]="$string"

done

for index in ${!array[@]}; do

     echo "$index is a ${array[$index]}"

done

LUB997 · 05-03-2010, 04:20 PM

I really like the idea of using the associative arrays and that is really the idea I was originally aiming for, but both my server that will in the end be running the script and my laptop which I am writing and testing the script on have Bash version 3.2.39, so I'll probably have to learn to use the hash method since I don't have Bash 4.0.

Kenhelm · 05-03-2010, 05:42 PM

You might have a carriage return character in $individual.
It can be caused by using files in Linux which have been edited in Windows.

Demonstration :-

Code:

individual=1234$'\r'    # $'\r' is a carriage return character.
lineoftext=abcdef

name[$individual]=$lineoftext
")syntax error: operand expected (error token is "

echo $individual $lineoftext
 abcdef

LUB997 · 05-03-2010, 06:19 PM

You were absolutely right about the carriage return being a huge part of the problem. I changed the line that said

lineoftext=`echo $lineoftext | sed "s/\@\ INDI// g"`;

to say

lineoftext=`echo $lineoftext | sed "s/\@\ INDI\r// g"`;

and suddenly there is no error message.

Funny how things seem so complicated and it usually turns out to be something so simple... Thank you so much for that suggestion! So simple, and I never would have thought of it. And of course the .ged file did originally come from a Windows environment since that is what runs the Family Tree Maker software that created it. I had converted it from Western enconding to UTF-8 so it would work with Bash in Linux, but never thought about carriage returns.

Now I can probably use the first example that David gave to set up arrays. The second example he gave won't work for me since I don't have Bash v4, but I tried the first example, and it worked flawlessly with my version of Bash, so I think I can set things up that way.

I'll play around with it a bit and let everyone know what I come up with and how it works out. I'm thinking problem solved though.

LUB997 · 05-03-2010, 06:37 PM

Problem is officially solved.

New code that works correctly:

Code:

#!/bin/bash
# Shell script to generate bygdebok sections by location from gedcom files formatted in UTF-8.

inputfile=test.ged;
category=0;
family=0;
declare -i individual;
declare -a name;

while read lineoftext
do

  if [[ ${lineoftext:0:1}    == "0" ]]
  then
    category=0;

  else
    if [[ ${lineoftext:0:1}    == "1" ]]
    then
      category=1;

    else
      if [[ ${lineoftext:0:1}    == "2" ]]
      then
        category=2;
      fi
    fi
  fi

  if [ $category == 0 ]
  then
    if [[ $lineoftext == *\@\ INDI* ]]
    then
      lineoftext=`echo $lineoftext | sed "s/0\ \@I// g"`;
      lineoftext=`echo $lineoftext | sed "s/\@\ INDI\r// g"`;
      lineoftext="$(echo $lineoftext | sed 's/0*//')";
      individual=$lineoftext;

    else
      if [[ $lineoftext == *\@\ FAM* ]]
      then
        lineoftext=`echo $lineoftext | sed "s/0\ \@F// g"`;
        lineoftext=`echo $lineoftext | sed "s/\@\ FAM// g"`;
        family=$lineoftext;
      fi
    fi
  fi

  if [ $category == 1 ]
  then
    if [[ $lineoftext == *NAME* ]]
    then
      lineoftext=`echo $lineoftext | sed "s/1\ NAME\ // g"`;
      lineoftext=`echo $lineoftext | sed "s/\/// g"`;
      name[$individual]=$lineoftext;
      echo $individual" "${name[$individual]};
    fi
  fi

done < $inputfile

exit 0

New output

Code:

./generate.sh

Code:

...
29 Milford Joel Houge
30 Ethel Mae Moore
31 Peter Oley Houge
32 Cecelia Berthe Hallsteinsdotter
33 Hallstein Petersson Utstono
34 Anna Lisbet Torbjørnsdotter Førland
35 Ola Sjursson Houge
36 Valborg Johannesdotter Mehus
37 Johannes Anfinnsson Indre Bauge
38 Siri Johannesdotter Ytre Færavik
39 George Meyers Moore
40 Emma Florence Barger
41 Sjur Ørjansson Hauge
42 Siri Torbjørnsdotter Øyjordo
43 Torbjørn Sjursson Houge
44 Madela Olsdotter Tjella
... and on and on for a very long time until the end of file (individual 1,039)

This output test demonstrates that the variables are now being properly and accurately stored and retrieved in an array using a variable as the array index, and that is what I was after, so I am considering the problem solved.

All suggestions were helpful, good suggestions, but the two suggestions that ended up solving the problem were David the H.'s first code example about how to do arrays with variables as the index and Kenhelm's idea to check for carriage returns. Thanks so much!

David the H. · 05-04-2010, 08:47 AM

Glad I could help you out. Those pesky windows line-endings always seem to throw things for a loop, don't they?

One comment on your code. I think you're relying too much on if-then statements. Bash has a couple of variations that can make your code cleaner. First there's "if-elif-else"

Code:

if [[ ${lineoftext:0:1} == "0" ]]; then
     "category=0"

elif [[ ${lineoftext:0:1} == "1" ]]; then
     "category=1"

elif [[ ${lineoftext:0:1} == "2" ]]; then
     category=2

else echo "Error!  Bad input."
     exit 1

fi

It would probably be better here to use a case statement, however.

Code:

case "${lineoftext:0:1}" in

     0) category=0 ;;

     1) category=1 ;;

     2) category=2 ;;

     *) echo "Error!  Bad input."
        exit 1  ;;

esac

But in this case it looks like you could even eliminate the branching entirely and simply use...

Code:

category=${lineoftext:0:1}

And perhaps just throw in a quick sanity test after it to make sure the value is within acceptable bounds.

catkin · 05-04-2010, 08:54 AM

Quote:

Originally Posted by David the H.

But in this case it looks like you could even eliminate the branching entirely and simply use...

Code:

category=${lineoftext:0:1}

+1 to that. grail suggested it in the second post of the thread.