LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
Search this Thread
Old 12-23-2008, 01:54 PM   #1
MikeyCarter
Member
 
Registered: Feb 2003
Location: Orangeville
Distribution: Fedora
Posts: 448

Rep: Reputation: 31
Question How do I identify what process are making up the WA statistic?


I have a WA% between 50-80%.

So I have a few questions.

1. Is that WA% only disk I/O or can it include network IO?
2. How do I identify which processes are making up that WA statistic?
3. Why does iostat report 100% utilization when the drive read/write speeds are under 1MB/s?

The reason I'm asking.

My Oracle stats are saying that the drives are running at about 5MB/s right now, the WA is 55%, and the iostat -m -x is reporting the drives running at 100% utilization with a read/write rate under 1MB/s. The read speed of the drive is 98MB/s (tested with hdparm -tT when I installed the drives)

I know I have a few queries using full tables scans which I'm working on but I'd think that 100% utilization should be running at the full 98MB/s.

I want to try to identify which process (most likely and oracle one) is using the 100% utilization/55% WA.
 
Old 12-23-2008, 04:15 PM   #2
thecarpy
Member
 
Registered: Apr 2005
Location: France
Distribution: Debian Squeeeeze
Posts: 116

Rep: Reputation: 18
WA% is IO, it can be disk and/or network IO. Anything the CPU is waiting on can cause increased WA%.

Most of time, from what I have experienced, it occurs with swapping. Since the drive is apparently sort of idle, you should look at the network, however.

Code:
tcpdump -i eth0
will show you the network activity, look for ever re-occurring patterns.
Code:
lsof -i
might help you find out which processes are using the network.
Code:
netstat -i TCP
is also pretty good, if you know what is doing what with the network on your computer.
 
Old 12-23-2008, 04:32 PM   #3
thecarpy
Member
 
Registered: Apr 2005
Location: France
Distribution: Debian Squeeeeze
Posts: 116

Rep: Reputation: 18
Sorry, I forgot, it may not be network related. Have you got ext3? How much free space do you have on the drives? The ext3 fs is very bad in performance once the drives start to fill up.

Do you not have the possibility to create indexes in the Oracle database, full table scans are expensive and should be avoided at all costs, especially if the table is large...

How much swap is used?

You should see that in
Code:
top
,
Code:
swapon -s
or
Code:
free
?

Which process is using most swap:

in top, press O and the o, this will order processes by virtual memory usage.


Oh, damn, I forgot the most important one here, vmstat. read its man page, no too long and very informative!

Last edited by thecarpy; 12-23-2008 at 04:37 PM.
 
Old 12-23-2008, 05:06 PM   #4
thecarpy
Member
 
Registered: Apr 2005
Location: France
Distribution: Debian Squeeeeze
Posts: 116

Rep: Reputation: 18
To see which applications use how much effective virtual memory, you look at the RES column in top.

You should watch out with Oracle processes, however, since they all include the SGA (system global area). You will have to deduct the size of that from the values in RES. It can be that once you have deducted that, the process using most RES will be further down the list, so watch out for that one....

Sorry for the many posts ...

Last edited by thecarpy; 12-23-2008 at 05:38 PM.
 
Old 12-23-2008, 09:16 PM   #5
MikeyCarter
Member
 
Registered: Feb 2003
Location: Orangeville
Distribution: Fedora
Posts: 448

Original Poster
Rep: Reputation: 31
I think I may have found the main culprit.


my setup is: two 98MB/s and two 61MB/s drives.
Code:
+==================+     +==================+
|   Dom0 - F8      |     |   DomU - OEL     |
|                  |     |                  |
|  VG- OraASM_01   |-----|  Raw: ASM Part 1 | 
|  VG- OraASM_02   |     |  Raw: ASM Part 2 |
|  VG- OraASM_03   |     |  Raw: ASM Part 3 |
|                  |     |                  |
+==================+     +==================+
I had a theory that it might be LVM2 slowing me down. So I cleared a 250GB (61 MB/s) drive. I'm in the process of re-balancing things to the new config. (This way the DB is on it's own drive)

What I'm seeing is the OraASM_01-OraASM_03 are running combined reads at 25MB/s max. iostats is showing 100% drive utilization. Oracle is writing to the new 250GB at about 10MB/s but is only showing 16% utilization.

I'm guessing here that LVM is doing something which is effectively throttling the Oracle ASM logical volumes. If this is the case it explains a lot of what I've been seeing. It would explain why none of my stats make sense.

Does anyone know what could be causing the throttling effect of LVM?

Also thanks for the network commands I'll try them out.

Last edited by MikeyCarter; 12-23-2008 at 09:18 PM.
 
Old 12-24-2008, 06:11 AM   #6
MikeyCarter
Member
 
Registered: Feb 2003
Location: Orangeville
Distribution: Fedora
Posts: 448

Original Poster
Rep: Reputation: 31
Ok I'm such a moron!

First the command I was looking for is iotop. However seems to only work with new kernels and a newer python.


Now to my problem. iostats has been screaming my problem at me and I was ignoring it because I didn't understand. It seems that my system/drives can only handle ~100 io requests per second. This is why I'm seeing my drive speed only working at .01 MB/s and 100% utilization. I thought it was LVM because it was doing large read/writes while re-balancing the disks before.

So my first option is to create 4 x 75G lvm partitions on the 4 hard drives. This will give me ~400 requests per second. Second option is to get Oracle to send less IO requests.

My new question is, what is limiting the io requests? Is it the control, drive, drivers, kernel? How do I find out?
 
  


Reply

Tags
high, memory, oracle, usage


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Making a service/process startup by itself? Flybye Linux - Newbie 9 07-01-2008 11:37 PM
how to identify a zombie process and control it after parent dies vijaya_svk Programming 1 01-21-2008 04:44 AM
Scripting the tea making process... pwc101 General 9 05-01-2007 06:58 AM
Which process is making DNS queries? TruckStuff Linux - Networking 4 11-29-2005 04:05 PM
making a process last beyond a session Winter-Storm Linux - Newbie 6 11-10-2005 08:38 PM


All times are GMT -5. The time now is 11:38 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration