Unzip + Piping

iago · 11-17-2004, 12:52 PM

I need to process about 10gb of zipped files with a very small Perl script I'm writing. The problem is, I can't seem to get it to pipe properly. The command I'm using is this:

$ unzip -p /mnt/cdrom/MyBigFile.zip | ProxyParser.pl

Right now, it's just printing everything to stdout:

Code:

#!/usr/bin/perl -w

foreach(<STDIN>)
{
  print "$_";
}

The problem is, it seems to be unzipping the entire file in memory and not sending it to my perl script. Because of the size, I'm running out of memory:

Code:

iago@Slayer:~$ dmesg | grep ProxyParser.pl
Out of Memory: Killed process 14423 (ProxyParser.pl).
Out of Memory: Killed process 14432 (ProxyParser.pl).
iago@Slayer:~$

This problem also happens on Windows, where I'm using a similar unzip program. My goal is for this to run on Windows, since that's what our server is running. But getting it to work right on Linux would be a good step forward.

Any help is appreciated!

Thanks,
-iago

<edit> I noticed that I can pipe it into "more" just fine, but not into my perl script. So Apparently, I can't use foreach(<STDIN>) to parse it in Perl. Any idea how I can do it properly?

Medievalist · 11-17-2004, 01:55 PM

I apologize in advance for posting without a solution... but I can give you a lead or two.

First off, I'm not sure windows implements a true pipe. DOS didn't, it ran the entire first process and stored the output, then ran the second program. This often led to storage exhaustion on DOS when the same program ran fine on linux or VMS (which both implement true pipes). You'd probably be better off developing your code on the target OS because of such non-obvious incompatibilities.

Second, by default perl is reading records separated by newlines. Enormously long lines which don't bother gzip at all will overflow perl's read buffer, generating errors. Perl's read behaviour is somewhat configurable, but it won't parse records on a regex like awk can, or read fixed size data blocks as readily as C, so perl might not be the best language for your task.

Good luck!
--Charlie

iago · 11-17-2004, 02:11 PM

Right now I have a working version in Java. But here are some timings of our various strategies:

VB: 8 hours/month of logs (unzipped)
Java: 1 hour/month of logs (zipped)
Perl: 5 seconds/day of logs (unzipped), which is about 10 minutes for a month.

The longest line I encountered is about 1200 characters. I'm not sure whether or not Perl can handle that much.

If Windows is actually storing it and running the program with that input, then doing it with Perl is hopeless. After we finish up the first 3 sections, which are urgent, we can look into maybe moving onto a Linux server. The problem is, I'm the only Linux "expert" they have here, and I'm only here on a term position.