I have a bottleneck...the system execution time is a matter of seconds...but the real time is over an hour in some cases.

I realize network latency could be beyond my control...but what about I/O? Is it simply a matter of setting autoflush and not printing to STDOUT?

How should I go about debugging this, and if any, what solutions exist for improving this bottleneck?

First of all, you seem somewhat reluctant to let us know what program you're trying to optimize--why is that? At least, you could have told us what it does--all we know is that it connects to the net and prints stuff on stdout. Does it write to the disk as well? Is it a streaming ascii-art video client? Is it rsync? Also, what language is the program you're optimizing written in?

Anyways, here's my suggestion: compile the program with profiling enabled and ask your profiler what's taking your program so damn long.

Hope this helps.

Sorry about that. I wanted to avoid publicly badmouthing Clearcase *oh well*.

It seems it's not limited to any one application, although Clearcase is the most visible as it's the most widely used. The scripts that automate the user interaction are written in Perl and call clearcase commands, ie - checkout, rebase, deliver (and perl's times function returns only a few seconds for parent/child system times, neglibile at this point, while the overall start->finish time can be drastically higher. One operation takes around 30 seconds of system time, but the overall user experience is more than 15 minutes).

My guess, and I'm not too familiar with this, is that it's a combination of network connectivity, (although the pipe was tested at 32MB/s), and system I/O.

A colleague mentioned to me using sar and io-stat to debug the I/O...but I'm not familiar with interpreting the results.

Basically, I'm more curious in analyzing the I/O, as several tools unrelated to clearcase seem to have this issue. General overall performance seems slow.

try timing it with the bash builtin `time', to get system, user and wall-clock time, and see if anything looks odd. Also, for disk I/O, having a look at hdparm may be a good idea. Otherwise, I'm pretty much out of ideas, having never worked with clearcase, and "general overall performance" being a bit... big to work with.

bash's time is the same thing as perl's 'times' function, which I am logging, so that wouldn't tell me anything I don't already know.

what should i do with hdparm, from what I gather it's a util for manipulating the harddrive parameters.

possibly -f (flushing the buffer)?

You could start by writing a short Perl prog to run on the server and check cpu & disk I/O perf. If no problem, implies network issue, in which case, run the same prog from your desktop against server.
A v useful perl module is: http://search.cpan.org/~bloonix/Linu.../Statistics.pm
Also, just looking at top should give you an idea as to whether it's local to the server or not.