[perl scripting] help with applying array in a specific scenario

confusednewbie · 04-17-2011, 02:18 PM

Hi Linux forumers,

I have been having this particular problem for a while and hope you can help. I have basic knowledge of arrays in Perl, but I can't adapt the text book examples to my particular situation. Any scripting pointers will help greatly, thanks!

For my situation, for example, I have a reference text file called "File1" that contains the following lines (tab delimited):

Quote:

identiy id_index copynum chr start end confidence
JANE 1 2 1 51598 1500664 212.19
JANE 1 1 1 1617778 1662463 10.0
JANE 1 2 1 1677447 11503149 5427.0
JANE 1 3 1 11509172 11536855 0.29
JANE 1 2 2 11538322 16889665 2840.0
JANE 1 3 2 16890117 16929044 4.51

I have a second text file, "File2" that contains the following lines (tab delimited):

Quote:

probe chr position
CN_466171 1 60500
CN_513370 1 72178
CN_502616 1 76204
CN_511519 1 839258
CN_502615 2 75799
CN_489778 2 11560400
CN_502614 2 16900500

GOAL: What I'm trying to use an array for is to read in the "chr", "start", "end" of each line from File1 and see if any of the lines from File2 ("chr", "position") falls within the range of "start" and "end" given that the "chr" values are equal.

LOGIC: For a line in File1, the "chr", "start", "end" values are read. Every line (corresponding to a "probe") in File2 is read and compared to the read line values from File1: If "chr" value of a probe in File2 matches "chr" value of that line in File1 AND the "position" value of the probe is >="start" and <="end" of that line in File1, then a +1 counter is added.

For each line of File1, this check is done for every line of File2 and the total number of probes (or lines) from File2 satisfying the above condition is counted and printed as a new column in that line; then the process repeats for the next line of File1.

Resulting in something like this (note that the numprobe corresponds to the total number of lines/probes in File2 that matched "chr" values and had its "position" value fall within the "start"/"end" values of the File1 line):

Quote:

identiy id_index copynum chr start end confidence numprobe
JANE 1 2 1 51598 1500664 212.19 4
JANE 1 1 1 1617778 1662463 10.0 0
JANE 1 2 1 1677447 11503149 5427.0 0
JANE 1 3 1 11509172 11536855 0.29 0
JANE 1 2 2 11538322 16889665 2840.0 1
JANE 1 3 2 16890117 16929044 4.51 1

Basically a search and compare of entirety of File2 for each line of File1; if a line of File2 satisfies the given conditions, then add +1 to counter for new "numprobe" column of File1, if not then do not add to counter and move to next line of File2. Process repeats for every line of File1, with counter reset at 0 at the start of each line for File1.

Thank you very much for any scripting pointers. It's frustrating because I know what I need to do in my mind, but it's very difficult to translate those thoughts into actual script.

smallpond · 04-17-2011, 04:12 PM

The most direct approach from your description is:

1) open files and check for errors

2) Read all of File2 into an array of arrays AoA. The two indices are row and field.

Code:

$row = 0;
while (<FILE2>) {
    $AoA[$row++] = [ split ];
}

3) Read File1 line by line.

Code:

while (<FILE1>) {
    @f1 = split;

4) For each line, scan through all rows of AoA performing your logic.

Code:

    foreach $R (@AoA) {
        if ($f1[3] eq $R->[1])   # compare chr
        etc.

If the files are huge or you need it to be more efficient, then you can optimize some things at the cost of more code. For example, a 3-level array (or a hash) would let you only look through the File2 values for the chr value that you want. Depends on your requirement.

Tinkster · 04-17-2011, 05:03 PM

Moved: This thread is more suitable in <PROGRAMMING> and has been moved accordingly to help your thread/question get the exposure it deserves.

grail · 04-18-2011, 01:50 AM

Not sure if it has to be perl, but maybe this awk will also give you ideas:

Code:

awk 'FNR==NR{if($2 ~ /[0-9]/)chr[$2]=chr[$2]?chr[$2]" "$3:$3;next}{split(chr[$4],arr);for(x in arr)if(arr[x] >= $5 && arr[x] <= $6)ctr++;print $0,ctr;ctr=0}' file2 file1

bigearsbilly · 04-18-2011, 04:36 AM

still need help or finished?

Juako · 04-18-2011, 10:26 AM

plain bash:

Code:

#!/bin/bash
line=0
while read identity id_index copynum chr start end confidence; do
        (( line++ == 0)) && continue
        line=0
        numprobe=0
        while read probe chr2 position; do
                (( line++ == 0)) && continue
                (( chr == chr2 && start <= position && position <= end && numprobe++ ))
        done < file2
        echo "$identity $id_index $copynum $chr $start $end $confidence $numprobe"
done < file1

The three lines of code referencing $line variable are there to skip reading the first line in the input files.

Of course, it can be optimized in various ways (using arrays would be one optimization). This is just to show the algoritm exactly as you stated it.

grail · 04-18-2011, 12:03 PM

Copied your code Juako and as is with the supplied inputs I get all sorts of errors flying up the screen:

Code:

./d.sh: line 6: ((: chr: expression recursion level exceeded (error token is "chr")
./d.sh: line 6: ((: chr: expression recursion level exceeded (error token is "chr")
./d.sh: line 6: ((: chr: expression recursion level exceeded (error token is "chr")
./d.sh: line 6: ((: chr: expression recursion level exceeded (error token is "chr")
./d.sh: line 6: ((: chr: expression recursion level exceeded (error token is "chr")
./d.sh: line 6: ((: chr: expression recursion level exceeded (error token is "chr")
./d.sh: line 6: ((: chr: expression recursion level exceeded (error token is "chr")
./d.sh: line 6: ((: chr: expression recursion level exceeded (error token is "chr")
identiy id_index copynum chr start end confidence 0
./d.sh: line 6: ((: position: expression recursion level exceeded (error token is "position")
JANE 1 2 1 51598 1500664 212.19 4
./d.sh: line 6: ((: position: expression recursion level exceeded (error token is "position")
JANE 1 1 1 1617778 1662463 10.0 0
./d.sh: line 6: ((: position: expression recursion level exceeded (error token is "position")
JANE 1 2 1 1677447 11503149 5427.0 0
./d.sh: line 6: ((: position: expression recursion level exceeded (error token is "position")
JANE 1 3 1 11509172 11536855 0.29 0
./d.sh: line 6: ((: position: expression recursion level exceeded (error token is "position")
JANE 1 2 2 11538322 16889665 2840.0 1
./d.sh: line 6: ((: position: expression recursion level exceeded (error token is "position")
JANE 1 3 2 16890117 16929044 4.51 1

I do see the correct data hidden amongst it but you might have a little checking to do

Juako · 04-18-2011, 12:16 PM

Quote:

Originally Posted by grail

Copied your code Juako and as is with the supplied inputs I get all sorts of errors flying up the screen:

that's odd... i run it with the OP's input and it works fine here. Maybe a different bash version? Its running in bash 4, but afaict not using bash 4 specific syntax. Anyway i'll try it in bash 3 and try to see what it is.

*edit* : working fine in another machine with bash 3 too
*edit1* : ahh there i been able to reproduce your error. I had ommited the "header" line in the input files in my tests. When i add the headers the read loops obviously fail trying to make arithmetic comparisons. I'm updating the code and fixing.
*edit2* updated