LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Other *NIX Forums > Other *NIX
User Name
Password
Other *NIX This forum is for the discussion of any UNIX platform that does not have its own forum. Examples would include HP-UX, IRIX, Darwin, Tru64 and OS X.

Notices


Reply
  Search this Thread
Old 03-14-2008, 04:14 PM   #1
b1f30
Member
 
Registered: Nov 2007
Location: GMT -5, who wants to know
Distribution: Gentoo, Debian, BSD, Slack
Posts: 119

Rep: Reputation: 15
HP-UX cryptic console errors


HP-UX cryptic console errors

Tuesday, March 18 2008 SOLVED! This was a *really* fun brain-bender of a problem to hack.

What happened? Previous admin or one of the current operators/developers got a bit zany on this HP-UX server and decided to wipe everything in the /usr/bin directory. Not good. Consoles would spawn, but you couldn't log in for obvious reasons.

I tried single user mode:

http://www.brandonhutchinson.com/Boo...user_mode.html

From there, I manually mounted volumes such as /usr and /var and /tmp as described below, but alas, I couldn't even use vim! It didn't exist! What the...?! Something is wrong with /usr/bin, or where commands are stored.

Code:
# ls -l /usr/bin
Nothing. Ugh.

Luckily, this server was dual boot. So, I took the option of booting into the working server, and mounting the broken OS volumes inside of it.

HP-UX has some curious naming conventions. The disk volumes for the good OS appear in /dev/vg00.

I had to create a new Volume Group so I could use the broken OS'es volumes. This is how HP-UX handles it's disks. You create the Volume Group, then you create the volumes themselves (partitions if you will - correct me if I'm wrong here). You can do all of this by cheating a bit using the 'smh' web tool. You submit 'smh' via command line, and if your HP-UX server has X forwarding (mine does), Firefox will pop up, and you get this web-based admin tool the likes I've never seen. From there, I was able to create a new Volume Group (vg01), and then assign the broken OS'es volumes (partitions) to it.

NOTE: Creating Volume Groups does *not* mount other partitions! It simply tells HP-UX 'Hey, there's another disk here we can do something with!'.

Now that my Volume Group is defined, what I did next was to 'fsck' the unmounted volumes to check for errors. Note that fsck'ing in HP-UX is a bit different than a Linux system.

Code:
bash# fsck -F vxfs /dev/vg01/lvol8
This produced nothing significant, so I went on to the next step: mount the bad volume(s), fix /usr/bin.

I created a temporary mount point:

Code:
# mkdir -pv /tmp/tmp_mnt
Then I mount the bad /usr volume:

Code:
# mount /dev/vg01/lvol8 /tmp/tmp_mnt
Now I've got the bad /usr mounted in the good server.

Luckily, I have a clone HP-UX server with the OS version I need sitting right on top the server I'm currently logged into.

Using GNOME, I do Places --> Connect to Server --> and open an ssh connection to the server mounting the bad /usr, and the server that has the /usr/bin files I need. Now I have two folders on my GNOME desktop. I drag-n-drop the files from the 'good' server (/usr/bin/*) and drop them into the mounted empty /usr/bin volume. Make sense?

In any case, this is a hackish approach, I don't recommend you do it this way, but it worked for me. The server is back online, so I can ssh and admin whatever's left.

--- ORIGINAL POST ---

All:

From Solaris to HP-UX to AIX. When did someone decide that this was a good career?

Anyway...

I've got this dual-boot HP-UX box:

HP-UX B.11.23 <-- Primary
HP-UX B.11.31 <-- Secondary

The hardware includes 2 HBA's, which I've disconnected for troubleshooting purposes.

11.31 boots fine, but 11.23 spits out some really really strange console errors - simply, it shows a service starting, then starts spitting periods '.' across the screen, filling the buffer for how many lines I can't tell. Then it goes to the next service, and the same thing happens. It takes an inordinate amount of time for anything to start/boot, then it just hangs - no login prompt.

Something like this:

Starting NFS:

.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
Mounting /vg0/00/x/x/x/x
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................
.................................................................


Idears?

Last edited by b1f30; 03-18-2008 at 11:18 AM.
 
Old 03-16-2008, 08:30 AM   #2
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,831
Blog Entries: 15

Rep: Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669
At the ISL> prompt during boot type:

hpux -is
Boots into single user mode. It shouldn't mount most filesystems or attempt to start most of the services. You can then manually mount things you'll need (/var, /usr and /tmp at a minimum) for most of the commands to work. You can then try to see what services should start and attempt them manually (the init scripts are in /sbin/init.d and /sbin/rc?.d for the run levels instead of /etc/init.d, /etc/rc?.d like Solaris and Linux).

You don't say what class of hardware you're on (e.g. RPxxxx for PA-RISC or RXxxxx for Itanium for most newer HP servers.)

P.S. This list isn't really a good one for HP-UX. Despite my asking for it the maintainers don't feel it necessary to put up an HP-UX sub-forum. You should post to the Dutchworks mailing list or to HP's own ITRC forums.

Info for Dutchworks:
To subscribe/unsubscribe contact majordomo@dutchworks.nl
Name: hpux-admin@dutchworks.nl Owner: owner-hpux-admin@dutchworks.nl
Archives: ftp.dutchworks.nl:/pub/digests/hpux-admin (FTP, browse only)
http://www.dutchworks.nl/htbin/hpsysadmin (Web, browse & search)


Info for ITRC:
itrc.hp.com
They'll make you register there.

Some really knowledgeable people on both the list and the forum but Dutchworks seems to have a lot less traffic than it did before the ITRC came along.
 
Old 03-17-2008, 07:52 AM   #3
b1f30
Member
 
Registered: Nov 2007
Location: GMT -5, who wants to know
Distribution: Gentoo, Debian, BSD, Slack
Posts: 119

Original Poster
Rep: Reputation: 15
Talking

Quote:
Originally Posted by jlightner View Post
At the ISL> prompt during boot type:

hpux -is
Boots into single user mode. It shouldn't mount most filesystems or attempt to start most of the services. You can then manually mount things you'll need (/var, /usr and /tmp at a minimum) for most of the commands to work. You can then try to see what services should start and attempt them manually (the init scripts are in /sbin/init.d and /sbin/rc?.d for the run levels instead of /etc/init.d, /etc/rc?.d like Solaris and Linux).
Was able to boot into single user mode. I'm going to try your suggestions mounting everything manually

Side note: I used the second OS install (disk 1) to manually mount and fsck the volumes on the original OS install (disk 0) which taught me how to fsck HP-UX volumes, but in the end served no real purpose. The filesystems did not appear damaged. That leads me to believe there is perhaps a service or kernel module that is making this system bail.

Quote:
Originally Posted by jlightner View Post
You don't say what class of hardware you're on (e.g. RPxxxx for PA-RISC or RXxxxx for Itanium for most newer HP servers.)
You are correct. What's a good one liner that will show that info? Thanks!

Quote:
Originally Posted by jlightner View Post
P.S. This list isn't really a good one for HP-UX. Despite my asking for it the maintainers don't feel it necessary to put up an HP-UX sub-forum. You should post to the Dutchworks mailing list or to HP's own ITRC forums.

Info for Dutchworks:
To subscribe/unsubscribe contact majordomo@dutchworks.nl
Name: hpux-admin@dutchworks.nl Owner: owner-hpux-admin@dutchworks.nl
Archives: ftp.dutchworks.nl:/pub/digests/hpux-admin (FTP, browse only)
http://www.dutchworks.nl/htbin/hpsysadmin (Web, browse & search)


Info for ITRC:
itrc.hp.com
They'll make you register there.

Some really knowledgeable people on both the list and the forum but Dutchworks seems to have a lot less traffic than it did before the ITRC came along.
Thanks for the info. I don't think LQ needs an HP-UX thread either, but it's nice to know there are people like yourself who are willing to advise even the biggest *NIX n00bs like myself. So cheers to that.

EDIT: HP-UX should *totally* have it's own thread. It's *NIX - which is what Linux is based on. :-p This site is very popular, and serves as a great non-book resource for admins like myself who are sort of forced into supporting/fixing random *NIX boxes before we go shell out the money for courses or books.

Last edited by b1f30; 03-17-2008 at 08:12 AM.
 
Old 03-17-2008, 10:35 AM   #4
b1f30
Member
 
Registered: Nov 2007
Location: GMT -5, who wants to know
Distribution: Gentoo, Debian, BSD, Slack
Posts: 119

Original Poster
Rep: Reputation: 15
UPDATE:

Wow - this box is HOSED. It's used by several developers/engineers for various tests.

After some digging, I've found that everything in /usr/bin has been swiped. Joy!
 
Old 03-17-2008, 11:17 AM   #5
b1f30
Member
 
Registered: Nov 2007
Location: GMT -5, who wants to know
Distribution: Gentoo, Debian, BSD, Slack
Posts: 119

Original Poster
Rep: Reputation: 15
UPDATE:

After copying un-modified files from a clone machine, I repopulated /usr/bin and now this machine boots like it's supposed to. You can imagine why this machine failed to boot properly.

EDIT: System in question:

Code:
bash-3.00# uname -a
HP-UX Q51103 B.11.23 U 9000/800 2381673951 unlimited-user license
 
Old 03-17-2008, 01:46 PM   #6
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,831
Blog Entries: 15

Rep: Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669
Sounds like you got a handle on it.

The "model" command will tell you the model and other things:

e.g.

For an Itanium server we have:
model
ia64 hp server rx1600
(rx1600 is the model - ia64 lets you know it is itanium)

For an PA_RISC server we have:
model
9000/800/rp3410
(rp3410 is the model, HP 9000 series 800 lets us know it is PA-RISC)

For a much older PA_RISC (running HP-UX 10.20 even):
model
9000/800/K580

By the way - I wasn't saying LQ wasn't a good resource but rather that it wasn't a good one for HP-UX questions. I've been on LQ for over 3 years and find it very helpful for Linux.

It just sort of irks me that my favorite UNIX is treated as a red headed step child. It probably wouldn't bother me if they didn't have Solaris and AIX forums which are the other two major commercial UNIX versions. That is to say if the site were ONLY Linux or ONLY OSS (including *BSD) I could understand it but obviously it isn't.

Anyway I've already raised that to the mods in the past. They're not interested and say they don't have a lot of requests for it. To me that's not a reasonable response. It's a bit like a grocery store saying "We don't carry that brand because we don't get a lot of requests for it." Obviously if the brand is important people will just go to a store that does sell it and if not you'll use the brands you see - that doesn't mean there's no desire for the brand but rather that people dealt with what was available.

Last edited by MensaWater; 03-17-2008 at 01:55 PM.
 
Old 03-18-2008, 09:23 AM   #7
b1f30
Member
 
Registered: Nov 2007
Location: GMT -5, who wants to know
Distribution: Gentoo, Debian, BSD, Slack
Posts: 119

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by jlightner View Post
Sounds like you got a handle on it.

The "model" command will tell you the model and other things:

e.g.

For an Itanium server we have:
model
ia64 hp server rx1600
(rx1600 is the model - ia64 lets you know it is itanium)

For an PA_RISC server we have:
model
9000/800/rp3410
(rp3410 is the model, HP 9000 series 800 lets us know it is PA-RISC)

For a much older PA_RISC (running HP-UX 10.20 even):
model
9000/800/K580
bash-3.00# model
9000/800/rp3440

Two of 'em sandwiched atop one another.

So far, so good. It's leftover stuff from a previous admin/lab operator.
 
Old 03-18-2008, 11:08 AM   #8
b1f30
Member
 
Registered: Nov 2007
Location: GMT -5, who wants to know
Distribution: Gentoo, Debian, BSD, Slack
Posts: 119

Original Poster
Rep: Reputation: 15
Marked as [SOLVED] - sorry - ripping of Gentoo forum's convention. :-)
 
  


Reply

Tags
console, hpux, serial


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Disable errors output to console Tushar Pednekar Linux - Software 2 07-17-2006 08:10 PM
cryptic errors compiling simplest C program code_noc Slackware 4 10-06-2003 08:38 PM
cryptic problem. Gantrep Linux - Software 5 04-20-2003 01:49 PM
Console Errors on boot Michael Ellis Programming 2 09-30-2002 02:48 PM
Cryptic error messages wizard23 Linux - General 3 01-03-2002 06:31 PM

LinuxQuestions.org > Forums > Other *NIX Forums > Other *NIX

All times are GMT -5. The time now is 06:40 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration