[SOLVED] Linux Shell script

charu · 03-29-2011, 11:07 AM

I am trying to create one large file from several small ones.

I have

$FILE1
$FILE2
$FILE3

And want to check if each exists then if it does create and/or append to a file called
$BIGFILE

I am doing this
fnAppend()
{
#create $BIGFILE using $FILE1
cat $FILE1 > $BIGFILE

#if $FILE2 exists append it to $BIGFILE
if [ -e $FILE2 ];
then
cat $FILE2 >> $BIGFILE
fi

#if $FILE3 exists append it to $BIGFILE
if [ -e $FILE3 ];
then
cat $FILE3 >> $BIGFILE
fi

}

I am getting odd results, such as $FILE1 is being appended to, and $BIGFILE is bigger than it should be.

I have tried various syntaxes to do this, but thought I would ask see if someone could give me the right way to do it without me having to spend any more hours experimenting, as for some reason i seem to be stuck and going round in circles with this, which i think should probably be quite simple, but i guess i am just tired!

pwc101 · 03-29-2011, 11:29 AM

Can't you just

Code:

cat FILE? > bigfile.txt

grail · 03-29-2011, 11:36 AM

Well apart from the fact that you never test if $FILE1 points to an actual file or that $BIGFILE does not already exist, my guess would be that in all the code you did not show us,
some of your variables may be set incorrectly.

Also, if you include [code][/code] tags around your code as pwc101 has, your indentation will be preserved and code easier to read.

charu · 03-29-2011, 12:34 PM

Ah sorry, yes the code tags.

i am actually not using variables, I just put them to save space,but it probably just confused what i was doing, so the actual code I am using is

Code:

fnSlHe()
{
## initially remove any current file as the function may be called more than once##
rm /path/to/file/12498.txt

cat /path/to/file/12498-1.txt > /path/to/file/12498.txt

if [ -e /path/to/file/12498-2.txt ];
then
cat /path/to/file/ >> /path/to/file/12498.txt
fi

if [ -e /path/to/file/12498-3.txt ];
then
cat /path/to/file/12498-3.txt >> /path/to/file/12498.txt
fi
}

If that looks correct then I guess it is something else in the script causing the problem rather than this function, if it does not look correct without the variables, then please let me know and I will search a bit deeper tomorrow after a good nights sleep.

pwc101 · 03-29-2011, 01:17 PM

Assuming your shell is bash:

Code:

fnSlHe(){
    # Get a list of the files we'll be iterating over
    files=(/path/to/files/12498-*.txt)

    # Use the first file's name to get a basename for the catted file
    prefix="${files[0]%-*}".txt

    # Check we don't already have a file of that name
    if [ -e "$prefix" ]; then
        printf "Warning: output file (%s) already exists. Aborting.\n" "$prefix"
        return 1
    else
        # OK, so now we need to iterate over all the files in $files, and
        # cat each file in $files out to $prefix.
        for ((i=0; i<${#files[@]}; i++)); do
            cat "${files[i]}" >> "$prefix"
        done
    fi
}

charu · 03-29-2011, 02:39 PM

Thanks very much, that is great, the iterating through a list of files with that pattern is a much better way of getting the existing files than the way I was trying to do it, much neater and much appreciated.

grail · 03-29-2011, 08:35 PM

Glad you got a solution, but in answer to why yours was not working and assuming there are no typos in post #4, the offending part is your second file test
and cat:

Code:

if [ -e /path/to/file/12498-2.txt ];
then
    cat /path/to/file/ >> /path/to/file/12498.txt # here you are catting the directory and not the second file
fi

charu · 03-31-2011, 06:59 AM

Yes it was a typo

I was too tired to be posting on forums that day!

The condensed script using the iteration works perfectly and is great.

I realised today that the problem I was having with my long winded method of trying to do this, was actually to do with something else, but I am glad I asked anyway as I prefer the condensed method and will in future use that for similar functions.

I was going on file size to check if the files had joined successfully. The files were much bigger than they should have been hence I thought it was a problem in this cat function....the file size was though due to a sed command prior to the cat...although my code to cat was pretty bad considering the new code i have now been given.

I was doing this search/replace...

Code:

sed -i 's/\|HeadingOne\|/\|HeadingTwo\|/g' $SLHD3

I was searching for a string containing a pipe, thinking that escaping the pipe with the backslash would be ok, but that was causing some weirdness, making the files huge, leading me to think that the cat command was joining files multiple times.

Anyway that part of the problem has was fixed by using double quotes round the sed command and no escape character for the pipe.

Code:

sed -i "s/|HeadingOne|/|HeadingTwo|/g" $SLHD3

Thankyou