[SOLVED] History of processes causing I/O wait

lpwevers · 03-01-2017, 08:45 AM

Quote:

Originally Posted by sundialsvcs

I suggest that you should simply – and, promptly – make the case to management that maximizing the amount of RAM in these systems (to the extent that the motherboards will support!) will have a direct and immediate business impact. Furthermore, "chips are cheap."

You should have a qualified engineer look at exactly how the memory-cards are now deployed in these machines and to construct a plan to optimize the amount of RAM that is available in the various systems with an efficient outlay of money. In any case, it won't be much money, and the bottom-line business impact is quite obviously there. Presumably, in order to prosecute your business, you need to "run those queries." Trying to do it without enough resources is: "penny-wise and pound-foolish." This case can very easily be made, and you can quote me on that.

Oh, trust, me, been there, done that. And then you know managers, business procedures, forms in triple, blah, blah, blah, loads of management crap. But you're right, told them again after this incident, so hopefully they'll listen this time. (It's now under management consideration).

I guess this is just one of those cases, where if they don't upgrade memory soon, I'll just end up saying something like, "told you so...." and then laugh in their face.

chrism01 · 03-03-2017, 06:08 AM

1. make sure you emailed them and cc'd yourself. Unfortunately sometimes they come looking for a scapegoat.
Never been had like that, but you hear stories.
2. Old SysAdmin saying "If you can afford it, it's not powerful enough" ; aka over time, it's normal evolution to keep increasing the load on a system until sooner or later it over-loads

Habitual · 03-03-2017, 06:35 AM

Quote:

Originally Posted by lpwevers

I've managed to setup more advanced monitoring using f.i. collectl and we managed to collect the culprit. We managed to trace it down to a certain user who had scheduled a job in the system to generate some reports over the night. Needless to say his query was less then optimally written.

What's "f.i." please?

lpwevers · 03-03-2017, 07:56 AM

Quote:

Originally Posted by Habitual

What's "f.i." please?

For Instance.
(As in for example)

lpwevers · 03-06-2017, 04:18 PM

Ah, just to let you guys know. I've gotten the new memory. From what I can tell now it looks promising. Fingers crossed.....

sundialsvcs · 03-06-2017, 05:46 PM

"Yeah, I thought so."

Now, you might wish to consider doing some other things, such as using a batch workload manager. For example, could those reports or long running jobs be run in unattended mode, such that you could control the "mix" of work that is being performed at any one time?

Also, see if you can use facilities such as ulimit, as well as any configuration files provided by the software, to "dole out" the memory resource so that everyone gets their fair share and no process can monopolize it.

In one past life, I created a very simple set of commands and invited the users to use them, which they did. These commands used something like the time command to measure(!) the execution time and resource consumption of the programs, writing the data to a public disk file. Some very simple statistical analysis of this file, after a few weeks, enabled me to tell these users things that they never knew about their own software. I gently introduced the idea of the resource availability on the system and tools such as ulimit, promising them that if they used these things, they could achieve predictable completion times ... all the time. Which they did, and which they did!