HP-UX cryptic console errors
 

HP-UX cryptic console errors

Tuesday, March 18 2008 SOLVED! This was a *really* fun brain-bender of a problem to hack.

What happened? Previous admin or one of the current operators/developers got a bit zany on this HP-UX server and decided to wipe everything in the /usr/bin directory. Not good. Consoles would spawn, but you couldn't log in for obvious reasons.

I tried single user mode:

http://www.brandonhutchinson.com/Boo...user_mode.html

From there, I manually mounted volumes such as /usr and /var and /tmp as described below, but alas, I couldn't even use vim! It didn't exist! What the...?! Something is wrong with /usr/bin, or where commands are stored.

Nothing. Ugh.

Luckily, this server was dual boot. So, I took the option of booting into the working server, and mounting the broken OS volumes inside of it.

HP-UX has some curious naming conventions. The disk volumes for the good OS appear in /dev/vg00.

I had to create a new Volume Group so I could use the broken OS'es volumes. This is how HP-UX handles it's disks. You create the Volume Group, then you create the volumes themselves (partitions if you will - correct me if I'm wrong here). You can do all of this by cheating a bit using the 'smh' web tool. You submit 'smh' via command line, and if your HP-UX server has X forwarding (mine does), Firefox will pop up, and you get this web-based admin tool the likes I've never seen. From there, I was able to create a new Volume Group (vg01), and then assign the broken OS'es volumes (partitions) to it.

NOTE: Creating Volume Groups does *not* mount other partitions! It simply tells HP-UX 'Hey, there's another disk here we can do something with!'.

Now that my Volume Group is defined, what I did next was to 'fsck' the unmounted volumes to check for errors. Note that fsck'ing in HP-UX is a bit different than a Linux system.

This produced nothing significant, so I went on to the next step: mount the bad volume(s), fix /usr/bin.

I created a temporary mount point:

Then I mount the bad /usr volume:

Now I've got the bad /usr mounted in the good server.

Luckily, I have a clone HP-UX server with the OS version I need sitting right on top the server I'm currently logged into.

Using GNOME, I do Places --> Connect to Server --> and open an ssh connection to the server mounting the bad /usr, and the server that has the /usr/bin files I need. Now I have two folders on my GNOME desktop. I drag-n-drop the files from the 'good' server (/usr/bin/*) and drop them into the mounted empty /usr/bin volume. Make sense?

In any case, this is a hackish approach, I don't recommend you do it this way, but it worked for me. The server is back online, so I can ssh and admin whatever's left.

--- ORIGINAL POST ---

All:

From Solaris to HP-UX to AIX. When did someone decide that this was a good career?

Anyway...

I've got this dual-boot HP-UX box:

HP-UX B.11.23 <-- Primary
HP-UX B.11.31 <-- Secondary

The hardware includes 2 HBA's, which I've disconnected for troubleshooting purposes.

11.31 boots fine, but 11.23 spits out some really really strange console errors - simply, it shows a service starting, then starts spitting periods '.' across the screen, filling the buffer for how many lines I can't tell. Then it goes to the next service, and the same thing happens. It takes an inordinate amount of time for anything to start/boot, then it just hangs - no login prompt.

Something like this:

Starting NFS:

.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
Mounting /vg0/00/x/x/x/x
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................


Idears?

At the ISL> prompt during boot type:

hpux -is
Boots into single user mode. It shouldn't mount most filesystems or attempt to start most of the services. You can then manually mount things you'll need (/var, /usr and /tmp at a minimum) for most of the commands to work. You can then try to see what services should start and attempt them manually (the init scripts are in /sbin/init.d and /sbin/rc?.d for the run levels instead of /etc/init.d, /etc/rc?.d like Solaris and Linux).

You don't say what class of hardware you're on (e.g. RPxxxx for PA-RISC or RXxxxx for Itanium for most newer HP servers.)

P.S. This list isn't really a good one for HP-UX. Despite my asking for it the maintainers don't feel it necessary to put up an HP-UX sub-forum. You should post to the Dutchworks mailing list or to HP's own ITRC forums.

Info for Dutchworks:
To subscribe/unsubscribe contact majordomo@dutchworks.nl
Name: hpux-admin@dutchworks.nl Owner: owner-hpux-admin@dutchworks.nl
Archives: ftp.dutchworks.nl:/pub/digests/hpux-admin (FTP, browse only)
http://www.dutchworks.nl/htbin/hpsysadmin (Web, browse & search)


Info for ITRC:
itrc.hp.com
They'll make you register there.

Some really knowledgeable people on both the list and the forum but Dutchworks seems to have a lot less traffic than it did before the ITRC came along.

Was able to boot into single user mode. I'm going to try your suggestions mounting everything manually

Side note: I used the second OS install (disk 1) to manually mount and fsck the volumes on the original OS install (disk 0) which taught me how to fsck HP-UX volumes, but in the end served no real purpose. The filesystems did not appear damaged. That leads me to believe there is perhaps a service or kernel module that is making this system bail.

You are correct. What's a good one liner that will show that info? Thanks!

Thanks for the info. I don't think LQ needs an HP-UX thread either, but it's nice to know there are people like yourself who are willing to advise even the biggest *NIX n00bs like myself. So cheers to that.

EDIT: HP-UX should *totally* have it's own thread. It's *NIX - which is what Linux is based on. :-p This site is very popular, and serves as a great non-book resource for admins like myself who are sort of forced into supporting/fixing random *NIX boxes before we go shell out the money for courses or books.

UPDATE:

Wow - this box is HOSED. It's used by several developers/engineers for various tests.

After some digging, I've found that everything in /usr/bin has been swiped. Joy!

UPDATE:

After copying un-modified files from a clone machine, I repopulated /usr/bin and now this machine boots like it's supposed to. You can imagine why this machine failed to boot properly.

EDIT: System in question:

Sounds like you got a handle on it.

The "model" command will tell you the model and other things:

e.g.

For an Itanium server we have:
model
ia64 hp server rx1600
(rx1600 is the model - ia64 lets you know it is itanium)

For an PA_RISC server we have:
model
9000/800/rp3410
(rp3410 is the model, HP 9000 series 800 lets us know it is PA-RISC)

For a much older PA_RISC (running HP-UX 10.20 even):
model
9000/800/K580

By the way - I wasn't saying LQ wasn't a good resource but rather that it wasn't a good one for HP-UX questions. I've been on LQ for over 3 years and find it very helpful for Linux.

It just sort of irks me that my favorite UNIX is treated as a red headed step child. It probably wouldn't bother me if they didn't have Solaris and AIX forums which are the other two major commercial UNIX versions. That is to say if the site were ONLY Linux or ONLY OSS (including *BSD) I could understand it but obviously it isn't. :mad:

Anyway I've already raised that to the mods in the past. They're not interested and say they don't have a lot of requests for it. To me that's not a reasonable response. It's a bit like a grocery store saying "We don't carry that brand because we don't get a lot of requests for it." Obviously if the brand is important people will just go to a store that does sell it and if not you'll use the brands you see - that doesn't mean there's no desire for the brand but rather that people dealt with what was available.

bash-3.00# model
9000/800/rp3440

Two of 'em sandwiched atop one another.

So far, so good. It's leftover stuff from a previous admin/lab operator.