LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 09-18-2009, 02:23 PM   #1
sharptech
Member
 
Registered: Nov 2006
Posts: 104

Rep: Reputation: 15
Process problem


Hello,

Been awhile since I have been on the forums, however good to be back!

I am not a newb in Linux, however this is a bit of a newb question :P

We have a server running Red Hat Enterprise and we have something that is hanging the system. One of the developers does an import to one of the databases and then it starts to crawl. The Top command is not showing anything (everything looks normal) so looking for a way to try and track this to find out what is going on.

Let me know if there is any program I can install or any ideas you guys have to try and track this down.

Thanks
 
Old 09-18-2009, 03:30 PM   #2
anomie
Senior Member
 
Registered: Nov 2004
Location: Texas
Distribution: RHEL, Scientific Linux, Debian, Fedora
Posts: 3,935
Blog Entries: 5

Rep: Reputation: Disabled
There are a variety of tools (top, vmstat, netstat, ps) that can possibly help troubleshoot this at the system level. I generally prefer to install the sysstat package and then let sar run indefinitely. (The rpm should set up a cronjob that collects statistics for you.) A quick google turned up this primer if you're interested: http://www.linux.com/archive/feature/52570

It might be an even better idea to start troubleshooting this at the database level, though. Surely it's doing some specific complaining when things slow down.
 
Old 09-18-2009, 03:54 PM   #3
sharptech
Member
 
Registered: Nov 2006
Posts: 104

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by anomie View Post
There are a variety of tools (top, vmstat, netstat, ps) that can possibly help troubleshoot this at the system level. I generally prefer to install the sysstat package and then let sar run indefinitely. (The rpm should set up a cronjob that collects statistics for you.) A quick google turned up this primer if you're interested: http://www.linux.com/archive/feature/52570

It might be an even better idea to start troubleshooting this at the database level, though. Surely it's doing some specific complaining when things slow down.
Yeah I will def. take a look at that - thanks.

It does seem that this is happening at the database level which I am going to talk to the DB admin and see what we can try to figure out.

Thanks for the help.
 
Old 09-18-2009, 10:00 PM   #4
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 15,997

Rep: Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219
RHEL should (will) have sysstat pre-installed. All the history data you need will already be there.
I'd be thinking an I/O problem - have a look at iostat which is also a part of sysstat.
 
Old 09-21-2009, 08:29 AM   #5
sharptech
Member
 
Registered: Nov 2006
Posts: 104

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by syg00 View Post
RHEL should (will) have sysstat pre-installed. All the history data you need will already be there.
I'd be thinking an I/O problem - have a look at iostat which is also a part of sysstat.
OK I took a look at the systat logs on the day we had the problem. It did not look too bad other than this part here.

15:20:01 CPU %user %nice %system %iowait %idle
15:30:01 all 0.01 0.00 0.04 49.98 49.97
15:30:01 0 0.01 0.00 0.05 99.94 0.00
15:30:01 1 0.01 0.00 0.04 0.01 99.95
15:40:01 all 0.01 0.00 0.04 49.98 49.97
15:40:01 0 0.01 0.00 0.04 99.95 0.00
15:40:01 1 0.01 0.00 0.04 0.02 99.93
15:50:01 all 0.08 0.00 0.07 49.92 49.92
15:50:01 0 0.14 0.00 0.07 99.79 0.00
15:50:01 1 0.03 0.00 0.07 0.06 99.85
16:00:01 all 0.02 0.00 0.05 49.98 49.95
16:00:01 0 0.02 0.00 0.05 99.93 0.00
16:00:01 1 0.01 0.00 0.05 0.03 99.91
Average: all 9.52 0.00 0.70 5.57 84.22
Average: 0 8.58 0.00 0.71 8.73 81.98
Average: 1 10.45 0.00 0.68 2.41 86.46

This block of time had a problem (most likely when the database was running.)

This confirms there is a problem, but need to now pinpoint what is causing the problem, or why when the db does an import the system is crawling.
 
Old 09-21-2009, 07:57 PM   #6
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.9, Centos 7.3
Posts: 17,417

Rep: Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397
How big is the import? There can be a lot of work for a db, especially if the import requires converting from say csv to internal db structure.
Indexing can cause issues if you are importing into a non-empty table. Its hard for it to keep on serving requests and re-index the table and/or re-balance the indexes.
May be split it into several small imports and queue them?
 
Old 09-22-2009, 12:39 AM   #7
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 15,997

Rep: Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219
Quote:
Originally Posted by sharptech View Post
... we have something that is hanging the system. One of the developers does an import to one of the databases and then it starts to crawl.
What "crawls" - just the db users or the whole system ?.
That's showing I/O wait - presumably the DB subsystem. If it's causing swap and/or interfering with other (general) I/O to the same disks, then you've got a general problem.
Look at I/O at the actual device level.
 
Old 09-22-2009, 09:40 AM   #8
sharptech
Member
 
Registered: Nov 2006
Posts: 104

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by syg00 View Post
What "crawls" - just the db users or the whole system ?.
That's showing I/O wait - presumably the DB subsystem. If it's causing swap and/or interfering with other (general) I/O to the same disks, then you've got a general problem.
Look at I/O at the actual device level.
The system itself hangs and we have to reboot it to clear it up.

chrism01 - Not sure on the size I will have to check that out.
 
Old 09-23-2009, 08:30 AM   #9
sharptech
Member
 
Registered: Nov 2006
Posts: 104

Original Poster
Rep: Reputation: 15
I checked on the size of the import's into the database and they are around 6-8gig. The file comes in as a zip file so compressed it is about 500MB, and then the script uncompresses the file and imports into the database.

The odd thing is the lockup on the server happens once every 4 or 5 times we do the import and then we have to reboot and start over again. The system hangs at the same point and that is when the zip file is brought in and ready to unzip it hangs, but again it is intermittent.
 
Old 09-23-2009, 08:13 PM   #10
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.9, Centos 7.3
Posts: 17,417

Rep: Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397
Could be disk space in that case. Unzipping a file requires a lot of temp disk space.
 
Old 09-24-2009, 12:15 PM   #11
sharptech
Member
 
Registered: Nov 2006
Posts: 104

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by chrism01 View Post
Could be disk space in that case. Unzipping a file requires a lot of temp disk space.
The directory where the data is has 180+G available.

How can I check how much temp disk space it uses?
 
Old 09-24-2009, 07:46 PM   #12
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.9, Centos 7.3
Posts: 17,417

Rep: Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397
Should be enough space, although you need to check each time you do this until you catch it just before it hangs. Maybe something else is happening at the same time.
Check the system logs around that time.
If(!) it's happening during the unzip, that means its not a DB issue.
Exactly at what stage does it hang?
If(!) it hangs during the unzip, could be a bad zip file; what exact cmds/os is the zip/gzip happening.
What exact cmds/os is the unzip/gunzip?
IOW, need the details of exactly at what stage it happens, how the files is created, how its 'unzipped' (if that's the failure point).
 
Old 09-25-2009, 09:55 AM   #13
sharptech
Member
 
Registered: Nov 2006
Posts: 104

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by chrism01 View Post
Should be enough space, although you need to check each time you do this until you catch it just before it hangs. Maybe something else is happening at the same time.
Check the system logs around that time.
If(!) it's happening during the unzip, that means its not a DB issue.
Exactly at what stage does it hang?
If(!) it hangs during the unzip, could be a bad zip file; what exact cmds/os is the zip/gzip happening.
What exact cmds/os is the unzip/gunzip?
IOW, need the details of exactly at what stage it happens, how the files is created, how its 'unzipped' (if that's the failure point).
On the server we have 7zip and that is how the file is compressed. The 7zip file is transferred to the server and then it is unzipped and then the files are imported to the db. As you mentioned we have plenty of space on the server to handle the data.

The system hangs right after the file is transferred and before it is unzipped. This only happens every once every 4-5 times we do it, so that is what is not making sense to me. The system itself has no load, only when we do this.

The system we are running is Red Hat Enterprise.

Any more info I can give let me know. I will try to monitor this next time they do the import and see if I can catch anything as well.

Thanks
 
Old 09-25-2009, 01:38 PM   #14
SharpyWarpy
Member
 
Registered: Feb 2003
Location: Florida
Distribution: Fedora 18
Posts: 862

Rep: Reputation: 90
Sounds to me like a bad hard drive. Try installing a known good secondary hard drive and import your zip file to that and see how it goes.
 
Old 09-27-2009, 10:05 PM   #15
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.9, Centos 7.3
Posts: 17,417

Rep: Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397
Quote:
The system hangs right after the file is transferred and before it is unzipped.
Just to be pedantic ( ) is the txfr completing, or does it not complete ie the txfr is hanging?
You say 7zip; is this (src) an MS system?
What's the cmd/protocol you use to txfr; which machine ctrls it ie push or pull?
Have you monitored top whilst the txfr is running?
How is the src file created? Does the creator ensure that the src file has been created/completed and closed properly before zipping and txfring?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Finding the Process ID of a Process While Initiating the Process senthilmuthiah Linux - Newbie 7 04-02-2009 11:37 AM
problem with binding process and than execute another process chapa Programming 0 08-27-2003 04:47 AM
Process problem minor Linux - General 2 05-18-2001 11:24 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 04:43 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration