LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 12-17-2007, 04:49 PM   #1
brokenpromises
Member
 
Registered: Jan 2005
Location: NZ
Distribution: Fedora / Debian
Posts: 98

Rep: Reputation: 21
Help with interpreting / analyzing log files


Hello,

I'm having a issue with certain services randomly turning themself off (mysql and postfix). I am wondering what could be causing this?

I went to /var/log/

Code:
root@apollo /var/log $ ls
apache2          dmesg.1.gz             mail.info.2.gz  mysql.log.3.gz
auth.log         dmesg.2.gz             mail.info.3.gz  mysql.log.4.gz
auth.log.0       dmesg.3.gz             mail.info.4.gz  mysql.log.5.gz
auth.log.1.gz    dmesg.4.gz             mail.info.5.gz  mysql.log.6.gz
auth.log.2.gz    dpkg.log               mail.info.6.gz  mysql.log.7.gz
auth.log.3.gz    dpkg.log.1             mail.log        news
auth.log.4.gz    dpkg.log.2.gz          mail.log.0      ntpstats
auth.log.5.gz    faillog                mail.log.1.gz   proftpd
auth.log.6.gz    fsck                   mail.log.2.gz   syslog
boot             httpd                  mail.log.3.gz   syslog.0
bootstrap.log     mail.warn       syslog.1.gz
btmp             kern.log               mail.warn.0     syslog.2.gz
btmp.1           kern.log.0             mail.warn.1.gz  syslog.3.gz
daemon.log       kern.log.1.gz          mail.warn.2.gz  syslog.4.gz
daemon.log.0     kern.log.2.gz          mail.warn.3.gz  syslog.5.gz
daemon.log.1.gz  lastlog                messages        syslog.6.gz
daemon.log.2.gz  lpr.log                messages.0      user.log
daemon.log.3.gz  mail.err               messages.1.gz   user.log.0
debug            mail.err.0             messages.2.gz   user.log.1.gz
debug.0          mail.err.1.gz          messages.3.gz   user.log.2.gz
debug.1.gz       mail.err.2.gz          mysql           uucp.log
debug.2.gz       mail.err.3.gz          mysql.err       wtmp
debug.3.gz       mail.info              mysql.log       wtmp.1
dmesg            mail.info.0            mysql.log.1.gz  xferlog
dmesg.0          mail.info.1.gz         mysql.log.2.gz
Which one am I supposed to be looking at? Theres several files along with a directory for mysql alone:

Code:
root@apollo /var/log $ ls -l | grep mysql
drwxr-s--- 2 mysql adm     4096 2007-12-17 21:36 mysql
-rw-r----- 1 mysql adm        0 2007-06-26 23:03 mysql.err
-rw-r----- 1 mysql adm        0 2007-12-17 06:26 mysql.log
-rw-r----- 1 mysql adm       20 2007-12-16 06:27 mysql.log.1.gz
-rw-r----- 1 mysql adm       20 2007-12-15 06:26 mysql.log.2.gz
-rw-r----- 1 mysql adm       20 2007-12-14 06:26 mysql.log.3.gz
-rw-r----- 1 mysql adm       20 2007-12-13 06:26 mysql.log.4.gz
-rw-r----- 1 mysql adm       20 2007-12-12 06:26 mysql.log.5.gz
-rw-r----- 1 mysql adm       20 2007-12-11 06:26 mysql.log.6.gz
-rw-r----- 1 mysql adm       20 2007-12-10 06:26 mysql.log.7.gz
Same for mail:

Code:
root@apollo /var/log $ ls -l | grep mail
-rw-r----- 1 root  adm    14882 2007-12-17 20:06 mail.err
-rw-r----- 1 root  adm    57772 2007-12-16 06:14 mail.err.0
-rw-r----- 1 root  adm     2938 2007-12-09 04:55 mail.err.1.gz
-rw-r----- 1 root  adm     2878 2007-12-02 06:08 mail.err.2.gz
-rw-r----- 1 root  adm     2835 2007-11-25 04:58 mail.err.3.gz
-rw-r----- 1 root  adm   438978 2007-12-17 21:48 mail.info
-rw-r----- 1 root  adm  1059383 2007-12-17 06:23 mail.info.0
-rw-r----- 1 root  adm    57457 2007-12-16 06:46 mail.info.1.gz
-rw-r----- 1 root  adm   152712 2007-12-15 06:26 mail.info.2.gz
-rw-r----- 1 root  adm   151085 2007-12-13 06:23 mail.info.3.gz
-rw-r----- 1 root  adm   136413 2007-12-11 06:26 mail.info.4.gz
-rw-r----- 1 root  adm   117986 2007-12-06 06:26 mail.info.5.gz
-rw-r----- 1 root  adm   126008 2007-12-04 06:19 mail.info.6.gz
-rw-r----- 1 root  adm   881279 2007-12-17 21:48 mail.log
-rw-r----- 1 root  adm   286773 2007-12-16 06:46 mail.log.0
-rw-r----- 1 root  adm     9601 2007-12-09 06:42 mail.log.1.gz
-rw-r----- 1 root  adm    13435 2007-12-02 06:45 mail.log.2.gz
-rw-r----- 1 root  adm    13164 2007-11-25 06:38 mail.log.3.gz
-rw-r----- 1 root  adm   141921 2007-12-17 21:46 mail.warn
-rw-r----- 1 root  adm   185730 2007-12-16 06:29 mail.warn.0
-rw-r----- 1 root  adm    10463 2007-12-09 04:55 mail.warn.1.gz
-rw-r----- 1 root  adm    19859 2007-12-02 06:17 mail.warn.2.gz
-rw-r----- 1 root  adm    12583 2007-11-25 06:26 mail.warn.3.gz
Any help appreciated
 
Old 12-17-2007, 04:58 PM   #2
poonippi
Member
 
Registered: Dec 2007
Distribution: Debain Fluxbox
Posts: 72

Rep: Reputation: 15
Postfix and MySQL may use your system's syslog facility to write messages to your log file. Because system logging differs on different platforms, you should review your system documentation to learn about syslog. Most likely, you can find where an applications logging occurs by checking your system's /etc/syslog.conf file.
 
Old 12-17-2007, 05:00 PM   #3
gilead
Senior Member
 
Registered: Dec 2005
Location: Brisbane, Australia
Distribution: Slackware64 14.0
Posts: 4,123

Rep: Reputation: 162Reputation: 162
The current log is the one without the number - e.g. mysql.log, mail.err, mail.info, mail.log, mail.warn. The mysql log is 0 byte, so it's not not much help - try looking in /var/log/syslog and /var/log/messages as well.

Can you post any related messages from the logs as well as give some more info about the symptoms? When the services turn themselves off, can you no longer connect? Does netstat report the ports as no longer listening?
 
Old 12-18-2007, 02:50 AM   #4
brokenpromises
Member
 
Registered: Jan 2005
Location: NZ
Distribution: Fedora / Debian
Posts: 98

Original Poster
Rep: Reputation: 21
mysql is turning itself off. Mail is as well, but mysql just turned itself of twice within an hour. I just issued /etc/init.d/mysql restart without thinking much.

Ah: Here is what /var/log/syslog shows:

Code:
Dec 18 07:14:23 apollo kernel: Mem-info:
Dec 18 07:14:23 apollo kernel: DMA per-cpu:
Dec 18 07:14:23 apollo kernel: cpu 0 hot: high 42, batch 7 used:5
Dec 18 07:14:23 apollo kernel: cpu 0 cold: high 14, batch 3 used:3
Dec 18 07:14:23 apollo kernel: DMA32 per-cpu: empty
Dec 18 07:14:23 apollo kernel: Normal per-cpu: empty
Dec 18 07:14:23 apollo kernel: HighMem per-cpu: empty
Dec 18 07:14:23 apollo kernel: Free pages:        1488kB (0kB HighMem)
Dec 18 07:14:23 apollo kernel: Active:27822 inactive:43 dirty:0 writeback:0 unstable:0 free:372 slab:2352 mapped:26 pagetables:566
Dec 18 07:14:23 apollo kernel: DMA free:1488kB min:1492kB low:1864kB high:2236kB active:111288kB inactive:172kB present:139264kB pages_scanned:20923 all_unreclaimable? no
Dec 18 07:14:23 apollo kernel: lowmem_reserve[]: 0 0 0 0
Dec 18 07:14:23 apollo kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Dec 18 07:14:23 apollo kernel: lowmem_reserve[]: 0 0 0 0
Dec 18 07:14:23 apollo kernel: Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Dec 18 07:14:23 apollo kernel: lowmem_reserve[]: 0 0 0 0
Dec 18 07:14:23 apollo kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Dec 18 07:14:23 apollo kernel: lowmem_reserve[]: 0 0 0 0
Dec 18 07:14:23 apollo kernel: DMA: 2*4kB 3*8kB 1*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1488kB
Dec 18 07:14:23 apollo kernel: DMA32: empty
Dec 18 07:14:23 apollo kernel: Normal: empty
Dec 18 07:14:23 apollo kernel: HighMem: empty
Dec 18 07:14:23 apollo kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Dec 18 07:14:23 apollo kernel: Free swap  = 0kB
Dec 18 07:14:23 apollo kernel: Total swap = 0kB
Dec 18 07:14:23 apollo kernel: Free swap:            0kB
Dec 18 07:14:23 apollo kernel: 34816 pages of RAM
Dec 18 07:14:23 apollo kernel: 0 pages of HIGHMEM
Dec 18 07:14:23 apollo kernel: 1211 reserved pages
Dec 18 07:14:23 apollo kernel: 12454 pages shared
Dec 18 07:14:23 apollo kernel: 0 pages swap cached
Dec 18 07:14:23 apollo kernel: 0 pages dirty
Dec 18 07:14:23 apollo kernel: 0 pages writeback
Dec 18 07:14:23 apollo kernel: 26 pages mapped
Dec 18 07:14:23 apollo kernel: 2352 pages slab
Dec 18 07:14:23 apollo kernel: 566 pages pagetables
Dec 18 07:14:23 apollo kernel: Out of Memory: Kill process 15119 (mysqld) score 31878 and children.
Dec 18 07:14:23 apollo kernel: Out of memory: Killed process 15119 (mysqld).
Dec 18 07:14:24 apollo mysqld_safe[16774]: Number of processes running now: 0
Dec 18 07:14:24 apollo mysqld_safe[16776]: restarted
Dec 18 07:14:27 apollo mysqld[16780]: InnoDB: Error: pthread_create returned 12
Dec 18 07:14:28 apollo mysqld_safe[16782]: ended
I presume I am running out of memory and need to install more into the said machine? Is there any way around this? I.e. swap space etc? Here is the output of df -H:

Code:
$ df -H
Filesystem             Size   Used  Avail Use% Mounted on
/dev/sda1              5.3G   2.5G   2.6G  50% /
tmpfs                   68M      0    68M   0% /lib/init/rw
tmpfs                   68M   4.1k    68M   1% /dev/shm
Any help appreciated!
 
Old 12-18-2007, 05:34 AM   #5
poonippi
Member
 
Registered: Dec 2007
Distribution: Debain Fluxbox
Posts: 72

Rep: Reputation: 15
To increase swap space you would need to resize your swap partition, although not sure that is the best solution.

Code:
/proc/meminfo
is one way to see the details of your memory.

Code:
cfdisk /dev/sda
will show your your partion sizes.
 
Old 12-18-2007, 06:04 AM   #6
brokenpromises
Member
 
Registered: Jan 2005
Location: NZ
Distribution: Fedora / Debian
Posts: 98

Original Poster
Rep: Reputation: 21
Thanks. I'm just wondering - this server is running LAMP, with postfix for mail, and probably hosting around 30 sites. Is it normal for such a server to take up 192 MB (incl 64MB swap space) in memory?

Seems a bit odd!
 
Old 12-18-2007, 01:33 PM   #7
gilead
Senior Member
 
Registered: Dec 2005
Location: Brisbane, Australia
Distribution: Slackware64 14.0
Posts: 4,123

Rep: Reputation: 162Reputation: 162
The line Free swap = 0kB does say that you've run out of swap. I may be wrong but I thought that the line Total swap = 0kB meant that you didn't actually have any swap space. What does the output of the free command show you?

I don't know what you're serving up to know whether it's using too much memory so I'd suggest monitoring it. If it happens quickly and often, you could use top and ps. If it takes a while I'd wrap them in a script and capture their output or try and use the SysStat utilities to gather some performance stats over a period of time and see what's eating up the RAM.
 
Old 12-18-2007, 03:00 PM   #8
poonippi
Member
 
Registered: Dec 2007
Distribution: Debain Fluxbox
Posts: 72

Rep: Reputation: 15
You could also enabled slow-query logging in mysql, this reveals queries that take over a defined period of time to execute.

Mytop could also be useful. It's top for mysql.

192MB doesn't sound like nearly enough, I would definitely recommend upgrading (just my personal opinion :-)
 
Old 12-18-2007, 11:00 PM   #9
brokenpromises
Member
 
Registered: Jan 2005
Location: NZ
Distribution: Fedora / Debian
Posts: 98

Original Poster
Rep: Reputation: 21
Code:
 $ free
             total       used       free     shared    buffers     cached
Mem:        131224     103168      28056          0       1804      30024
-/+ buffers/cache:      71340      59884
Swap:            0          0          0
According to this I have no swap? What is the problem here? Long time ago I installed quota on this box (it is a web hosting box, and the application that controls disk space etc needed this) Could this be the problem?

Here is my fstab:

Code:
 $ cat /etc/fstab
/dev/sda1       /               ext3    defaults,errors=remount-ro,,usrquota,grpquota 0       1
/dev/sda2       none            swap    sw              0       0
 
Old 12-18-2007, 11:48 PM   #10
gilead
Senior Member
 
Registered: Dec 2005
Location: Brisbane, Australia
Distribution: Slackware64 14.0
Posts: 4,123

Rep: Reputation: 162Reputation: 162
I don't know what the sw option is for in your swap entry. If you run swapon -a as root from the command line, do you get any errors?
If you do, try changing the line to the following and running it again:
Code:
/dev/sda2  none  swap  defaults  0  0
 
Old 12-19-2007, 11:08 PM   #11
brokenpromises
Member
 
Registered: Jan 2005
Location: NZ
Distribution: Fedora / Debian
Posts: 98

Original Poster
Rep: Reputation: 21
Another question - the server in question is a VPS.

Therefore there is no "sda1" or "sda2" in /dev/

Code:
$ ls /dev/
agpgart   console  full   i2c-5    loop0  loop6    midi01  mixer       null  ram1   ram15  ram6    rmidi1     smpte1   tty1  tty7
audio     core     i2c-0  i2c-6    loop1  loop7    midi02  mixer1      port  ram10  ram16  ram7    rmidi2     smpte2   tty2  tty8
audio1    dsp      i2c-1  i2c-7    loop2  MAKEDEV  midi03  mixer2      ptmx  ram11  ram2   ram8    rmidi3     smpte3   tty3  tty9
audio2    dsp1     i2c-2  initctl  loop3  mem      midi1   mixer3      pts   ram12  ram3   ram9    sequencer  sndstat  tty4  urandom
audio3    dsp2     i2c-3  kmem     loop4  midi0    midi2   mpu401data  ram   ram13  ram4   random  shm        tty      tty5  xconsole
audioctl  dsp3     i2c-4  log      loop5  midi00   midi3   mpu401stat  ram0  ram14  ram5   rmidi0  smpte0     tty0     tty6  zero
Will this cause a problem when I run the swap commands?
 
Old 12-19-2007, 11:57 PM   #12
gilead
Senior Member
 
Registered: Dec 2005
Location: Brisbane, Australia
Distribution: Slackware64 14.0
Posts: 4,123

Rep: Reputation: 162Reputation: 162
That's interesting. What do you get when you run cat /proc/partitions or fdisk -l?
 
Old 12-20-2007, 01:24 AM   #13
brokenpromises
Member
 
Registered: Jan 2005
Location: NZ
Distribution: Fedora / Debian
Posts: 98

Original Poster
Rep: Reputation: 21
Code:
$ cat /proc/partitions
major minor  #blocks  name

   8     2    1048576 sda2
   8     1    5242880 sda1
Code:
 $ fdisk -l
 $
(Nothing happens)
 
Old 12-20-2007, 05:50 AM   #14
gilead
Senior Member
 
Registered: Dec 2005
Location: Brisbane, Australia
Distribution: Slackware64 14.0
Posts: 4,123

Rep: Reputation: 162Reputation: 162
The fdisk -l command needs to be run as root to return data. I'd try running swapon /dev/sda2 as root. If it works I'd try modifying /etc/fstab as per the post earlier.
 
Old 12-21-2007, 02:52 AM   #15
brokenpromises
Member
 
Registered: Jan 2005
Location: NZ
Distribution: Fedora / Debian
Posts: 98

Original Poster
Rep: Reputation: 21
Code:
root@localhost ~ $ swapon -a
swapon: cannot stat /dev/sda2: No such file or directory
Also re the 'sw' option in the fstab, I checked another VPS from the same provider with the same OS that we are running, and it has the same entry, and it runs.

Do I have to contact my provider to fix this?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Analyzing .WAV files indienick Programming 4 08-16-2006 03:30 PM
Deleted /var/log/messages, can't log any files-iptables chingyenccy Linux - Newbie 7 02-27-2005 05:03 PM
can't mount windows shares; need help interpreting log soren625 Linux - Networking 6 10-01-2004 08:05 PM
Interpreting Snort log files and alerts epeus Linux - Security 6 10-21-2002 10:47 AM
Can log files be time stamped? (such as FTP login and transfer log files) bripage Linux - Networking 6 08-08-2002 11:55 PM


All times are GMT -5. The time now is 09:09 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration