I'm trying to install a new server and having performance issues with one of my applications. It's one of those brand new HP DL580 G7s, with 4 8-core CPUs, and 384GB (yes, gigabytes) of memory. The application we're using uses an ISAM database and, using "strace", I can see it does record locking through "fcntl". Also using "strace" I can see it's spending an enormous amount of time waiting for locks on one of the files it's accessing.

Now, I can understand why having multiple processes (on multiple CPUs) competing for the same lock would make the machine run slower than its potential. What I can't understand is why this machine is slower than a five-year-old AIX system with four CPUs. One operation in particular, that takes 1-2 seconds on the AIX system, takes as long as 3 minutes on our new Linux machine. (Naturally, it's an interactive process, so our users actually have to sit there and wait for those 3 minutes.) Again, most of the time is spent waiting for locks on that one file.

Originally thinking it was a memory bandwidth issue, I disabled all but 8 cores, all of which are on the same CPU, but that didn't help much. We were also running some fairly heavy-duty reports on the system at the time, and I would have expected them to slow down, too, but they were fine. I'm also inclined to believe it's not an IO issue, since 384GB is more than enough to contain our entire database.

The only other thing I can remember to mention is that we're running RHEL 5.5, fully up-to-date.

Can anyone shed any light on why we're having this problem and what I might do to fix it?

Thanks

Depending on how you're using your database it might be best to switch that table to innodb that allows row level locking. What happens in ISAM is that when a record needs to be update (and for particular reads) the entire table is locked blocking other queries from completing (even though you have plenty of resources to handle this). With innodb you only lock the particular row in question rather than the whole table enabling you to have a much larger concurrency for writes and particular reads that would otherwise lock the entire table.

Regards,

Alunduil

We're not using a database engine. It's a proprietary, old-school ISAM database. The actual application is written in a language I doubt anyone here has heard of (Databus). Imagine a cross between assembly language and Cobol, with some ISAM database access statements built in.

Unfortunately, it's a third party application, and I don't have access to the source. Otherwise, I'd fix it myself.

Out of curiosity what configuration are the drives for this server in?

Regards,

Alunduil

RAID 5. It seems like the best configuration, since reads from our database significantly outnumber writes. Is that really relevant, though, given the amount of memory in the server?

Depends on how your buffers are setup but since it's a custom application I'm not sure where you would begin poking.

Regards,

Alunduil

Well, I know for sure, there is no explicit buffering in the high-level application code. There may be some buffering in the lower-level interpreter code, but I doubt it. From everything I've heard, I'm pretty sure the interpreter is just using Linux read(2) and write(2) system calls, directly - relying on Linux to handle all the buffering.