Hi. I'm jon.404, a Unix/Linux/Database/Openstack/Kubernetes Administrator, AWS/GCP/Azure Engineer, mathematics enthusiast, and amateur philosopher. This is where I rant about that which upsets me, laugh about that which amuses me, and jabber about that which holds my interest most: *nix.
The little instance that couldn't
Kids, gather around so I can tell you a story. A story of heartbreak, and frustration, and 'OhMahGerdWutTehF00kzRurThinking!'
I got a case yesterday where a user couldn't ssh to his EC2 instance. Simple enough, I see these quite often and I just about have a playbook to run by for these. *just about*, I say.
Yesterday I spent a decent amount of time troubleshooting with the user. Mostly emails back and forth (which would have gone faster, but the user and I both were wrapped up in other projects as well). By late evening, I'd given him a foolproof solution (foolproof, haha, guess that means I'm the fool!) and he said he'd get back to me today about it. Today started off with him saying the login failures weren't resolved. Sigh.
I asked him if we could do a screensharing session, because this issue, this is strange. I'm thinking selinux, perhaps? Maybe facls? No, perhaps it is something more insidious! PAM failures are being logged, but whatever could the problem be?!? I finally admitted that I would need to login to the instance myself to review the configuration. He shared the AMIs with my account and I got to work.
First, I added "PermitRootLogin yes" to the instance's sshd_config (after detaching the root volume and using a rescue instance to edit the file). Once I could login as root, I got to troubleshooting. Hrmmph. Kerberos, LDAP, and PAM, oh my!
I spent a few hours digging around trying to find a problem with the Kerberos/LDAP/PAM configuration, but in the end I was left with nothing. Frustrated, I tried to su to the user and edit the authorized_keys file. I couldn't su - to the user. Permission denied. See, *now* we're getting somewhere.
I figured I'd start at the top and work my way down...so I "ls -lh /"'d and my jaw dropped. They'd set permissions to 0700, owner root (of course). This explains why root could login, but authorized_keys wouldn't work for the user account (even though it was the same exact authorized_keys file!). I hang my head in shame. I let the desire for bigger and better problems cost me 2 hours of productivity. C'est la vie...dammit.
Perhaps it is time to say adios to Linux and go work in networking or something.
I got a case yesterday where a user couldn't ssh to his EC2 instance. Simple enough, I see these quite often and I just about have a playbook to run by for these. *just about*, I say.
Yesterday I spent a decent amount of time troubleshooting with the user. Mostly emails back and forth (which would have gone faster, but the user and I both were wrapped up in other projects as well). By late evening, I'd given him a foolproof solution (foolproof, haha, guess that means I'm the fool!) and he said he'd get back to me today about it. Today started off with him saying the login failures weren't resolved. Sigh.
I asked him if we could do a screensharing session, because this issue, this is strange. I'm thinking selinux, perhaps? Maybe facls? No, perhaps it is something more insidious! PAM failures are being logged, but whatever could the problem be?!? I finally admitted that I would need to login to the instance myself to review the configuration. He shared the AMIs with my account and I got to work.
First, I added "PermitRootLogin yes" to the instance's sshd_config (after detaching the root volume and using a rescue instance to edit the file). Once I could login as root, I got to troubleshooting. Hrmmph. Kerberos, LDAP, and PAM, oh my!
I spent a few hours digging around trying to find a problem with the Kerberos/LDAP/PAM configuration, but in the end I was left with nothing. Frustrated, I tried to su to the user and edit the authorized_keys file. I couldn't su - to the user. Permission denied. See, *now* we're getting somewhere.
I figured I'd start at the top and work my way down...so I "ls -lh /"'d and my jaw dropped. They'd set permissions to 0700, owner root (of course). This explains why root could login, but authorized_keys wouldn't work for the user account (even though it was the same exact authorized_keys file!). I hang my head in shame. I let the desire for bigger and better problems cost me 2 hours of productivity. C'est la vie...dammit.
Perhaps it is time to say adios to Linux and go work in networking or something.
Total Comments 2
Comments
-
I had a similar (except I was the stupid one) debugging problem yesterday too. I *thought* I knew what the application error message said. I thought it said the application couldn't create an TLS channel to the host. Later I was reviewing the host log for something else and discovered that the connection was totally fine, but that my user didn't have the correct permissions. So then I thought, "Why does the app error message say the connection is failing?" I went back to the application error message and actually READ it and discovered that it said my user didn't have the correct permissions. D'oh.Posted 03-19-2015 at 09:02 AM by vmccord 
-
I hate it when that happens. I can't tell you how many times I've debugged an issue for a few hours, only to discover once it is late enough that I couldn't possibly reconfigure it, that I overlooked one idiotically minor issue. It's pretty frustrating, I tell ya!
Posted 03-21-2015 at 08:18 PM by rocket357
