LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 03-08-2013, 07:37 AM   #1
leejayd
LQ Newbie
 
Registered: Mar 2013
Posts: 6

Rep: Reputation: Disabled
ssh issue with remote server


Hi,

I have a dedicated server to help me learn the ins and outs of Linux.

Just recently the web server went down returning 500 errors. I restarted the server but now the server doesn't respond to http.

I can ping the server fine but I can't access it at all via SSH. My provider allows me to boot to a recovery kernel. This post is concerned with the SSH issue. When I SSH to the recovery kernel I can mount the drives. All my data seems to be present. I've looked at the SSH and sshd config files but nothing seems a miss.

Could it be possible the boot is reaching run level that doesn't start SSH?

I`m running centos 5.7.

Any ideas for the next steps to troubleshoot SSH?
 
Old 03-08-2013, 08:03 AM   #2
shivaa
Senior Member
 
Registered: Jul 2012
Location: Grenoble, Fr.
Distribution: Sun Solaris, RHEL, Ubuntu, Debian 6.0
Posts: 1,800
Blog Entries: 4

Rep: Reputation: 286Reputation: 286Reputation: 286
Can you ssh, (since you write you can and mount drives)?

Code:
~$ service sshd status
And also check:
Code:
~$ ssh -p <port> <server>

Last edited by shivaa; 03-08-2013 at 08:04 AM.
 
Old 03-08-2013, 08:04 AM   #3
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,417

Rep: Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974
if it doesn't respond to ssh OR http then somethign bigger must be going on. Check /var/log/messages after a boot attempt. if it gets far enough through the boot process (i.e. mounts the actual disk) then all the boot up logs from the first few milliseconds should be in there to look through, and also /var/log/secure should show when ssh is coming up and down. I'd wonder if it's not the actual network config itself though.
 
Old 03-08-2013, 10:27 AM   #4
leejayd
LQ Newbie
 
Registered: Mar 2013
Posts: 6

Original Poster
Rep: Reputation: Disabled
I can only ssh into the recovery kernel. Hence i can't run commands against my own system. It looks like I have to mount the disks and manually fix files.

Is there a pattern I can use in a grep command as the files are huge.

Lee
 
Old 03-08-2013, 12:05 PM   #5
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,417

Rep: Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974
if you can mount the drives, you can reach the files. So just look at the last 1000 lines or so.
 
Old 03-09-2013, 08:53 AM   #6
leejayd
LQ Newbie
 
Registered: Mar 2013
Posts: 6

Original Poster
Rep: Reputation: Disabled
I've checked the secure log and found lots of repeating errors similar to :

loginuid
mar 5 01:10:01 ks31335 crond[7060]: pam_loginuid(crond:session): set_loginuid failed opening
loginuid
mar 5 01:10:50 ks31335 pure-ftpd: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
mar 5 01:10:50 ks31335 pure-ftpd: (?@127.0.0.1) [INFO] Logout.
mar 5 01:11:01 ks31335 crond[7120]: pam_loginuid(crond:session): set_loginuid failed opening
loginuid

I booted the server on 6th march and was able to ping but nothing of that date is in the secure file. Strange indeed. I wonder if there is another log file I should be looking at.
 
Old 03-09-2013, 09:13 AM   #7
shivaa
Senior Member
 
Registered: Jul 2012
Location: Grenoble, Fr.
Distribution: Sun Solaris, RHEL, Ubuntu, Debian 6.0
Posts: 1,800
Blog Entries: 4

Rep: Reputation: 286Reputation: 286Reputation: 286
Can you check log files inside /var/log/httpd directory. As you said above, you're getting 500 error, so search for it.

Code:
~$ cat access_log
~$ cat error_log
 
Old 03-09-2013, 09:24 AM   #8
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,599

Rep: Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241Reputation: 1241
That list of errors you had look like (I'm going to say IS) a blanket chmod gone wild (like a "chmod -R 755 *"). This would remove various privileges that are assigned to files to allow them to work properly. If you accidentally did this, the easiest solution is to reinstall - but do try to backup any data you want to keep first.
 
Old 03-09-2013, 10:02 AM   #9
leejayd
LQ Newbie
 
Registered: Mar 2013
Posts: 6

Original Poster
Rep: Reputation: Disabled
Thumbs up

The error log for httpd has this at the end :
PHP Code:
tailcannot open `error.log' for reading: No such file or directory
root@rescue:~/d1/var/log/httpd# tail error_log
[Sun Mar 03 04:02:08 2013] [notice] Digest: done
[Sun Mar 03 04:02:09 2013] [warn] RSA server certificate wildcard CommonName (CN) 
`*.lxlabs.com' does NOT match server name!?
[Sun Mar 03 04:02:09 2013] [notice] Apache/2.2.22 (Unix) DAV/2 PHP/5.2.17 mod_ssl/2.2.22 OpenSSL/0.9.8e-fips-rhel5 configured -- resuming normal operations
[Mon Mar 04 03:59:41 2013] [notice] caught SIGTERM, shutting down
[Mon Mar 04 03:59:41 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Mar 04 03:59:41 2013] [warn] RSA server certificate wildcard CommonName (CN) `*.lxlabs.com' 
does NOT match server name!?
[
Mon Mar 04 03:59:41 2013] [noticeDigestgenerating secret for digest authentication ...
[
Mon Mar 04 03:59:41 2013] [noticeDigestdone
[Mon Mar 04 03:59:42 2013] [warnRSA server certificate wildcard CommonName (CN) `*.lxlabs.com' does NOT match server name!?
[Mon Mar 04 03:59:42 2013] [notice] Apache/2.2.22 (Unix) DAV/2 PHP/5.2.17 mod_ssl/2.2.22 OpenS
SL/0.9.8e-fips-rhel5 configured -- resuming normal operations
root@rescue:~/d1/var/log/httpd# 
It was on the 4th march that I started to get problems. I've not chmodded any files. Is it possible this is a hack gone wrong?

I feel if I can get SSH working I'll be able to repair the web server.

Lee
 
Old 03-09-2013, 10:20 AM   #10
leejayd
LQ Newbie
 
Registered: Mar 2013
Posts: 6

Original Poster
Rep: Reputation: Disabled
I also checked the dmesg log and it looks like there are HD problems.


rtc_cmos 00:02: setting system clock to 2013-01-07 05:08:58 UTC (1357535338)
input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
ata1: SATA link down (SStatus 0 SControl 300)
ata2: SATA link down (SStatus 0 SControl 300)
md: Waiting for all devices to be available before autodetect
md: If you don't use raid, use raid=noautodetect
md: Autodetecting RAID arrays.
md: Scanned 0 and added 0 devices.
md: autorun ...
md: ... autorun DONE.
EXT3-fs (hda1): recovery required on readonly filesystem
EXT3-fs (hda1): write access will be enabled during recovery
kjournald starting. Commit interval 5 seconds
EXT3-fs (hda1): orphan cleanup on readonly fs
ext3_orphan_cleanup: deleting unreferenced inode 24491
ext3_orphan_cleanup: deleting unreferenced inode 24487
ext3_orphan_cleanup: deleting unreferenced inode 24486
ext3_orphan_cleanup: deleting unreferenced inode 24485
ext3_orphan_cleanup: deleting unreferenced inode 24484
ext3_orphan_cleanup: deleting unreferenced inode 24483
EXT3-fs (hda1): 6 orphan inodes deleted
EXT3-fs (hda1): recovery complete
EXT3-fs (hda1): mounted filesystem with writeback data mode
VFS: Mounted root (ext3 filesystem) readonly on device 3:1.
Freeing unused kernel memory: 528k freed
udevd (1734): /proc/1734/oom_adj is deprecated, please use /proc/1734/oom_score_adj instead.
eth0: Media Link On 100mbps full-duplex
md: Autodetecting RAID arrays.
md: Scanned 0 and added 0 devices.
md: autorun ...
md: ... autorun DONE.
EXT3-fs (hda1): using internal journal
kjournald starting. Commit interval 5 seconds
EXT3-fs (hda3): using internal journal
EXT3-fs (hda3): mounted filesystem with writeback data mode
Adding 1048572k swap on /dev/hda2. Priority:-1 extents:1 across:1048572k
 
Old 03-09-2013, 10:38 AM   #11
leejayd
LQ Newbie
 
Registered: Mar 2013
Posts: 6

Original Poster
Rep: Reputation: Disabled
I have a feeling my server is only reaching run level 1. Httpd and sshd which both don't work are at run level 2.

The plot thickens.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
SSH Issue - REMOTE HOST IDENTIFICATION HAS CHANGED! Rohit_4739 Linux - Security 7 03-15-2013 01:26 PM
Help with Ubuntu server remote ssh and local network ssh issues using putty. scottpops Linux - Server 8 05-17-2012 06:07 PM
remote ssh issue with the iptables command neteng996 Programming 1 09-21-2010 12:05 PM
SSH - Remote aplications in remote server Urien Linux - Newbie 11 04-04-2009 05:02 AM


All times are GMT -5. The time now is 06:17 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration