Reading 2 files and process both at same time
I have a perl script which takes a file as input, read it and process it.
Can I give 2 files as input and process them at same time rather than process them one after another means when the perl script starts it forks itself and process the two files with different child at same time. |
does this demostration help you:
Code:
[schneidz@hyper ~]$ touch hello world |
What do you really want to do? Merge two or more files?
|
Quote:
e.g. one file is storage related, one file is server related and each file has thousands of entry in it. The perl script read each file line by line and process them. What I want to do is, if I am merging the content, it will take huge time to process the other category of file as the processing is serial. So I categorized two different files and want to process them in parallel manner. As I said earlier, in someway to fork the script for two child script and each child will process separate files paralelly. And can this be done from inside the script or outside the script ?? |
Quote:
|
^ my previous suggestion was to just run it twice at the same time (calling different input files).
edit: ^ thats a weird requirement... that limitation mite be in place so that 1 person doesnt spam the cpu. not knowing the source of your script it would be hard to edit it but maybe you can have a function that is called twice with 2 different inputs. |
Quote:
So as per your idea the whole file processing programme in the script will be in one function and it will be called for each file ..right ??? |
i got another idea... if your script can take input from a pipe, maybe you can feed your script like so:
Code:
(cat file.1 & cat file.2) | divya.pl |
@OP: So your question is: How to fork in perl-script? Tried perldoc -f fork yet?
Edit: I have just found and old example-code of mine. (Be vareful, it's quite dusty.) Code:
#!/usr/local/bin/perl -w |
NevemTeve, thanks for the nice example. I will try that in my perl script and post the response.
|
Quote:
Because the script is behaving differently with presence and absence of both. Sometimes both childs started at same time and ended at same time, sometimes one child start & exit and then other child start & exit. On which basis this behaviour is changing ? |
The children run as they want (can), the only thing you can do for them is waiting for them to terminate before exiting the parent.
Let's not forget that the child-processes (or the child and the parent) are supposed to do unrelated jobs, otherwise there is no point in using them. PS: I should have used two wait's for the two children. Code:
$pid= wait; |
I modified the childs to read files from argument as below:
Code:
#!/usr/bin/perl |
That's what debugging is good for...
Code:
#!/usr/bin/perl |
In any shell:
Code:
perl script-one.pl & If you have many files to process, check out the -p numprocs argument of the xargs command. A general notion of Unix-ish systems is that commands ought to be simple, and fairly self-centered. Then, you get extra mileage out of them by simple shell features like these, and by "piping" multiple commands together so that the output of one becomes the input to another. It's a "disarmingly simple" idea, but it greatly reduces the complexity. Yes, "Perl can do anything you want." But maybe you can remove the complexity of "parallelism" from the program (regardless of language used), and move it up to the shell. |
All times are GMT -5. The time now is 12:33 PM. |