[SOLVED] Installed wrong bash patch package on remote server, can no longer log in

mattca · 09-25-2014, 01:08 PM

So I really screwed up.

I installed the 32 bit patch on a 64 bit remote system. wget still works, and I was able to download the correct package, but most commands simply respond with "/usr/bin/bash: no such file or directory". I realized I could use wget to download a replacement bash binary from another system I have, but I had to go to a meeting and my connection timed out. Trying to reconnect returns the above error, and then disconnects me.

What are my options???

It's a linode, if it matters.

thirdm · 09-25-2014, 01:40 PM

ssh takes an optional command as an argument. If you have another shell on the machine maybe you could login that way? e.g. ssh hades.oops.net /usr/bin/ksh

The -t argument may be of interest. Otherwise you won't see a prompt.

mattca · 09-25-2014, 01:59 PM

Thanks for the response. It appears that ssh runs the commands you give it through the default shell. So, even if I had another shell installed (I don't think I do), it would need to be invoked by a working bash

mattca · 09-25-2014, 02:22 PM

Fixed it!

Reboot into linode's rescue mode, ssh in via lish, mount the drive (it was in /dev/disk/by-uuid), then scp a working bash binary into the correct place. Reboot, done

ruario · 09-25-2014, 02:45 PM

Nice of you to follow up on your own post with a solution as it could be useful for anyone else who finds themselves in this position.

mattca · 09-25-2014, 02:58 PM

There have been many times that I've googled an issue, and the top response is one of my LQ threads from the last time I had the problem. Now I make it a habit to update with solutions when I find them

Alien Bob · 09-26-2014, 07:03 AM

I did the same thing yesterday... upgrading 32-bit bash with a 64-bit package on a remote server.
Luckily I still had a root prompt open, so what I did was change the "/bin/sh" symlink from /bin/bash to /bin/ash. Ash is compatible with the packagetools.
Then I ran "upgradepkg bash-4.1.012-i486-1_slack13.37.txz" and got my bash working again.
Finally, I reset the /bin/sh symlink to point to/bin/ bash again.

Problem fixed.

What you can do if you lost your root prompt before you ran into the issue: this is how you can get a root session with ash as the shell instead of bash:

Code:

$ su - -c ash
Please enter your OWN password as authentication.
(Enter your own password)
Password: 
\u@\h:\w\$ id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk)
\u@\h:\w\$

And from there you can do the same steps as described above (change the symlink, run upgradepkg, change the symlink back).
(note that in this case the root prompt will not be "#" but you are still root!)

Eric

kikinovak · 09-26-2014, 07:45 AM

Is there some bad alignment of the stars in the air? Last week I decided to switch mirrors on a public Slackware server and replaced the german mirror with the swiss mirror (swiss is good, always). I ran slackpkg update && slackpkg upgrade-all and wondered about the number of security updates. I watched slackpkg download a sh*tload of packages, and just before it had finished downloading, I realized I must have accidentally selected slackware-current, which meant I was about to install current on a production server. I did the equivalent of a basejump on my keyboard to hit Ctrl-C just seconds before the upgrade process was about to begin.

Phew.

ruario · 09-26-2014, 07:56 AM

Quote:

Originally Posted by Alien Bob

Ash is compatible with the packagetools.

I did not know this, good to find this out.

P.S. For others reading this. If you have local access to the machine, you could boot from install media, mount your disk partitions under /mnt, set ROOT=/mnt, then use upgradepkg to upgrade to the correct version.

ruario · 09-26-2014, 08:00 AM

Quote:

Originally Posted by kikinovak

basejump on my keyboard to hit Ctrl-C just seconds before the upgrade process was about to begin.

So Slackware is like basejumping, it makes life exciting! Or at least that is my takeaway from your story!

kikinovak · 09-26-2014, 09:04 AM

Quote:

Originally Posted by ruario

So Slackware is like basejumping, it makes life exciting! Or at least that is my takeaway from your story!

Yeah, I like the excitement. I drive a bike, I love climbing, I use Slackware.

Paulo2 · 09-26-2014, 11:01 AM

My first bet was like thirdm (post#2) I tested here with a virtual machine,
but once I lost the login shell I couldn't login at all.

Without another admin account, or sudo, is there a way to fix this remotely?
Or can be fixed only with physical access to the machine.

mostlyharmless · 09-26-2014, 02:58 PM

Oddly enough, I installed the 32 bit bash patch on my arm system, which I would have thought would lose my shell, but apparently I was able to repair it without any difficulties. Just lucky....

ruario · 09-27-2014, 01:04 AM

Quote:

Originally Posted by Paulo2

Without another admin account, or sudo, is there a way to fix this remotely?
Or can be fixed only with physical access to the machine.

mattca gave you an answer.

Quote:

Originally Posted by mattca

Reboot into linode's rescue mode, ssh in via lish, mount the drive (it was in /dev/disk/by-uuid), then scp a working bash binary into the correct place. Reboot, done

Here is an alternative.

Most VPS (like Linode, which the OP/mattca uses) have a recovery distro you can boot into. You could use this to run removepkg (to get rid of the old bash package) and then installpkg to install the new bash package, after mounting your disk and setting $ROOT appropriately.

You will need to patch installpkg and removepkg slightly due to differences between Slackware and other distros. First remove the -U and -l options from the tar extraction in installpkg, as their meanings have changed in later versions of tar. You could also remove the check for tar-1.13 to avoid waiting 5 seconds. Finally swap the order of the '-type f' and '-maxdepth 1' find options in removepkg.

Here is a patch:

Code:

--- installpkg.orig     2011-04-22 22:50:43.000000000 +0200
+++ installpkg  2014-09-27 06:15:05.000000000 +0200
@@ -107,22 +107,7 @@
 
 # So that we know what to expect...
 umask 022
-TAR=tar-1.13
-$TAR --help 1> /dev/null 2> /dev/null
-if [ ! $? = 0 ]; then
-  TAR=tar
-fi
-if [ ! "$(LC_MESSAGES=C $TAR --version)" = "tar (GNU tar) 1.13
-
-Copyright (C) 1988, 92,93,94,95,96,97,98, 1999 Free Software Foundation, Inc.
-This is free software; see the source for copying conditions.  There is NO
-warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
-
-Written by John Gilmore and Jay Fenlason." ]; then
-  echo "WARNING: pkgtools are unstable with tar > 1.13."
-  echo "         You should provide a \"tar-1.13\" in your \$PATH."
-  sleep 5
-fi
+TAR=tar
 
 usage() {
  cat << EOF
@@ -548,7 +533,7 @@
     cat $DESCRIPTION | grep "^$shortname:" >> $ADM_DIR/packages/$shortname 2> /dev/null
   fi
   echo "FILE LIST:" >> $ADM_DIR/packages/$shortname
-  ( cd $ROOT/ ; $packagecompression -dc | $TAR -xlUpvf - ) < $package >> $TMP/$shortname 2> /dev/null
+  ( cd $ROOT/ ; $packagecompression -dc | $TAR -xpvf- ) < $package >> $TMP/$shortname 2> /dev/null
   if [ "$(cat $TMP/$shortname | grep '^\./' | wc -l | tr -d ' ')" = "1" ]; then
     # Good.  We have a package that meets the Slackware spec.
     cat $TMP/$shortname >> $ADM_DIR/packages/$shortname
--- removepkg.orig      2009-04-28 22:44:26.000000000 +0200
+++ removepkg   2014-09-27 06:15:05.000000000 +0200
@@ -142,8 +142,8 @@
 # removed when ROOT= is used:
 cat_except() {
   ( cd "$1" && \
-    if [ $(find . -type f -maxdepth 1 | wc -l) -ne 1 ]; then
-      cat $(find . -type f -maxdepth 1 | grep -v "$2")
+    if [ $(find . -maxdepth 1 -type f | wc -l) -ne 1 ]; then
+      cat $(find . -maxdepth 1 -type f | grep -v "$2")
     fi
   )
 }