Sorry, I forgot, it may not be network related. Have you got ext3? How much free space do you have on the drives? The ext3 fs is very bad in performance once the drives start to fill up.
Do you not have the possibility to create indexes in the Oracle database, full table scans are expensive and should be avoided at all costs, especially if the table is large...
How much swap is used?
You should see that in
Which process is using most swap:
in top, press O and the o, this will order processes by virtual memory usage.
Oh, damn, I forgot the most important one here, vmstat. read its man page, no too long and very informative!