HP-UX cryptic console errors
Tuesday, March 18 2008 SOLVED! This was a *really* fun brain-bender of a problem to hack.
What happened? Previous admin or one of the current operators/developers got a bit zany on this HP-UX server and decided to wipe everything in the /usr/bin directory. Not good. Consoles would spawn, but you couldn't log in for obvious reasons.
I tried single user mode:
http://www.brandonhutchinson.com/Boo...user_mode.html
From there, I manually mounted volumes such as /usr and /var and /tmp as described below, but alas, I couldn't even use vim! It didn't exist! What the...?! Something is wrong with /usr/bin, or where commands are stored.
Nothing. Ugh.
Luckily, this server was dual boot. So, I took the option of booting into the working server, and mounting the broken OS volumes inside of it.
HP-UX has some curious naming conventions. The disk volumes for the good OS appear in /dev/vg00.
I had to create a new Volume Group so I could use the broken OS'es volumes. This is how HP-UX handles it's disks. You create the Volume Group, then you create the volumes themselves (partitions if you will - correct me if I'm wrong here). You can do all of this by cheating a bit using the 'smh' web tool. You submit 'smh' via command line, and if your HP-UX server has X forwarding (mine does), Firefox will pop up, and you get this web-based admin tool the likes I've never seen. From there, I was able to create a new Volume Group (vg01), and then assign the broken OS'es volumes (partitions) to it.
NOTE: Creating Volume Groups does *not* mount other partitions! It simply tells HP-UX 'Hey, there's another disk here we can do something with!'.
Now that my Volume Group is defined, what I did next was to 'fsck' the unmounted volumes to check for errors. Note that fsck'ing in HP-UX is a bit different than a Linux system.
Code:
bash# fsck -F vxfs /dev/vg01/lvol8
This produced nothing significant, so I went on to the next step: mount the bad volume(s), fix /usr/bin.
I created a temporary mount point:
Code:
# mkdir -pv /tmp/tmp_mnt
Then I mount the bad /usr volume:
Code:
# mount /dev/vg01/lvol8 /tmp/tmp_mnt
Now I've got the bad /usr mounted in the good server.
Luckily, I have a clone HP-UX server with the OS version I need sitting right on top the server I'm currently logged into.
Using GNOME, I do Places --> Connect to Server --> and open an ssh connection to the server mounting the bad /usr, and the server that has the /usr/bin files I need. Now I have two folders on my GNOME desktop. I drag-n-drop the files from the 'good' server (/usr/bin/*) and drop them into the mounted empty /usr/bin volume. Make sense?
In any case, this is a hackish approach, I don't recommend you do it this way, but it worked for me. The server is back online, so I can ssh and admin whatever's left.
--- ORIGINAL POST ---
All:
From Solaris to HP-UX to AIX. When did someone decide that this was a good career?
Anyway...
I've got this dual-boot HP-UX box:
HP-UX B.11.23 <-- Primary
HP-UX B.11.31 <-- Secondary
The hardware includes 2 HBA's, which I've disconnected for troubleshooting purposes.
11.31 boots fine, but 11.23 spits out some really really strange console errors - simply, it shows a service starting, then starts spitting periods '.' across the screen, filling the buffer for how many lines I can't tell. Then it goes to the next service, and the same thing happens. It takes an inordinate amount of time for anything to start/boot, then it just hangs - no login prompt.
Something like this:
Starting NFS:
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
Mounting /vg0/00/x/x/x/x
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
Idears?