Hi all.

I'm running into this problem with one of my scripts. The script mangles a large amount of ASCII data (~400 MB). The script concatenates N files with arbitrary number of records (records are of fixed number of lines). This is piped to "awk", which inserts a separator line after a selected number of lines. This is at last piped to "csplit" which splits the STDIN on those separators.

Just in case you haven't figured it out, this is a scrpit to unify batch size for the input processing.

The script works flawlessly on Tru64 UNIX and ZSh 2.x

When run on Linux Mandrake 9.1 and ZSh 4.1.0 it beltches after certain number of output batches have been created. The error reported is:

csplit: memory exhausted

BASh 2.0.5b reports "broken pipe" on this scrpit.

When I take out the last pipe and redirect output of "cat ... | awk ..." into a file it works OK. When I feed that file to "csplit", again, it works. The only thing I can conclude is that that last pipe is causing problems.

I don't think that increasing pipe size would help. I've tried reducing the output batch size. It still fails, only after several more data has been done. The error is totally reproducable in my case.

Is there some issue with "csplit" that it doesn't handle pipes?

Nixie.

It looks like csplit is running out of memory, "crashing" as a result, and the pipe is therefore broken. I don't think the problem is the pipe itself, it's csplit running out of memory. It may work if you feed that file into split and not work in the full chain of commands because the other commands use memory too.

I'm not sure what to do; you could run "ulimit -a" and see if that's limiting the amount of memory you can use.

If I understand what you're doing, I'd be tempted to replace the whole thing with just plain split(1). It knows how to split a file or standard input into files with N lines in each. You could skip awk altogether.

Lyle