LinuxQuestions.org - ssh issue with remote server

- Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)

- - ssh issue with remote server (https://www.linuxquestions.org/questions/linux-newbie-8/ssh-issue-with-remote-server-4175453252/)

ssh issue with remote server

Hi,

I have a dedicated server to help me learn the ins and outs of Linux.

Just recently the web server went down returning 500 errors. I restarted the server but now the server doesn't respond to http.

I can ping the server fine but I can't access it at all via SSH. My provider allows me to boot to a recovery kernel. This post is concerned with the SSH issue. When I SSH to the recovery kernel I can mount the drives. All my data seems to be present. I've looked at the SSH and sshd config files but nothing seems a miss.

Could it be possible the boot is reaching run level that doesn't start SSH?

I`m running centos 5.7.

Any ideas for the next steps to troubleshoot SSH?

Can you ssh, (since you write you can and mount drives)?

Code:

~$ service sshd status

And also check:

Code:

~$ ssh -p <port> <server>

if it doesn't respond to ssh OR http then somethign bigger must be going on. Check /var/log/messages after a boot attempt. if it gets far enough through the boot process (i.e. mounts the actual disk) then all the boot up logs from the first few milliseconds should be in there to look through, and also /var/log/secure should show when ssh is coming up and down. I'd wonder if it's not the actual network config itself though.

I can only ssh into the recovery kernel. Hence i can't run commands against my own system. It looks like I have to mount the disks and manually fix files.

Is there a pattern I can use in a grep command as the files are huge.

Lee

if you can mount the drives, you can reach the files. So just look at the last 1000 lines or so.

I've checked the secure log and found lots of repeating errors similar to :

loginuid
mar 5 01:10:01 ks31335 crond[7060]: pam_loginuid(crond:session): set_loginuid failed opening
loginuid
mar 5 01:10:50 ks31335 pure-ftpd: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
mar 5 01:10:50 ks31335 pure-ftpd: (?@127.0.0.1) [INFO] Logout.
mar 5 01:11:01 ks31335 crond[7120]: pam_loginuid(crond:session): set_loginuid failed opening
loginuid

I booted the server on 6th march and was able to ping but nothing of that date is in the secure file. Strange indeed. I wonder if there is another log file I should be looking at.

Can you check log files inside /var/log/httpd directory. As you said above, you're getting 500 error, so search for it.

Code:

~$ cat access_log

~$ cat error_log

That list of errors you had look like (I'm going to say IS) a blanket chmod gone wild (like a "chmod -R 755 *"). This would remove various privileges that are assigned to files to allow them to work properly. If you accidentally did this, the easiest solution is to reinstall - but do try to backup any data you want to keep first.

The error log for httpd has this at the end :

PHP Code:




tail: cannot open `error.log' for reading: No such file or directory 
root@rescue:~/d1/var/log/httpd# tail error_log 
[Sun Mar 03 04:02:08 2013] [notice] Digest: done 
[Sun Mar 03 04:02:09 2013] [warn] RSA server certificate wildcard CommonName (CN) `*.lxlabs.com' does NOT match server name!? 
[Sun Mar 03 04:02:09 2013] [notice] Apache/2.2.22 (Unix) DAV/2 PHP/5.2.17 mod_ssl/2.2.22 OpenSSL/0.9.8e-fips-rhel5 configured -- resuming normal operations 
[Mon Mar 04 03:59:41 2013] [notice] caught SIGTERM, shutting down 
[Mon Mar 04 03:59:41 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) 
[Mon Mar 04 03:59:41 2013] [warn] RSA server certificate wildcard CommonName (CN) `*.lxlabs.com' does NOT match server name!? 
[Mon Mar 04 03:59:41 2013] [notice] Digest: generating secret for digest authentication ... 
[Mon Mar 04 03:59:41 2013] [notice] Digest: done 
[Mon Mar 04 03:59:42 2013] [warn] RSA server certificate wildcard CommonName (CN) `*.lxlabs.com' does NOT match server name!? 
[Mon Mar 04 03:59:42 2013] [notice] Apache/2.2.22 (Unix) DAV/2 PHP/5.2.17 mod_ssl/2.2.22 OpenS 
SL/0.9.8e-fips-rhel5 configured -- resuming normal operations 
root@rescue:~/d1/var/log/httpd#

It was on the 4th march that I started to get problems. I've not chmodded any files. Is it possible this is a hack gone wrong?

I feel if I can get SSH working I'll be able to repair the web server.

Lee

I also checked the dmesg log and it looks like there are HD problems.

rtc_cmos 00:02: setting system clock to 2013-01-07 05:08:58 UTC (1357535338)
input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
ata1: SATA link down (SStatus 0 SControl 300)
ata2: SATA link down (SStatus 0 SControl 300)
md: Waiting for all devices to be available before autodetect
md: If you don't use raid, use raid=noautodetect
md: Autodetecting RAID arrays.
md: Scanned 0 and added 0 devices.
md: autorun ...
md: ... autorun DONE.
EXT3-fs (hda1): recovery required on readonly filesystem
EXT3-fs (hda1): write access will be enabled during recovery
kjournald starting. Commit interval 5 seconds
EXT3-fs (hda1): orphan cleanup on readonly fs
ext3_orphan_cleanup: deleting unreferenced inode 24491
ext3_orphan_cleanup: deleting unreferenced inode 24487
ext3_orphan_cleanup: deleting unreferenced inode 24486
ext3_orphan_cleanup: deleting unreferenced inode 24485
ext3_orphan_cleanup: deleting unreferenced inode 24484
ext3_orphan_cleanup: deleting unreferenced inode 24483
EXT3-fs (hda1): 6 orphan inodes deleted
EXT3-fs (hda1): recovery complete
EXT3-fs (hda1): mounted filesystem with writeback data mode
VFS: Mounted root (ext3 filesystem) readonly on device 3:1.
Freeing unused kernel memory: 528k freed
udevd (1734): /proc/1734/oom_adj is deprecated, please use /proc/1734/oom_score_adj instead.
eth0: Media Link On 100mbps full-duplex
md: Autodetecting RAID arrays.
md: Scanned 0 and added 0 devices.
md: autorun ...
md: ... autorun DONE.
EXT3-fs (hda1): using internal journal
kjournald starting. Commit interval 5 seconds
EXT3-fs (hda3): using internal journal
EXT3-fs (hda3): mounted filesystem with writeback data mode
Adding 1048572k swap on /dev/hda2. Priority:-1 extents:1 across:1048572k

I have a feeling my server is only reaching run level 1. Httpd and sshd which both don't work are at run level 2.

The plot thickens.