Hi There,
Am having Oracle Linux running on VMware, but every day the server stack or shutdown cant access to the servers until we restart the box which now has to be a while.
This is not a good health that the server does stay for some days without restarting.
went to the logs and I found the following errors what could be the problem with my servers?

Other servers run the Oracle RAC, which also affect the services to my customers.

Nov 19 11:16:33 ptzntwb2 kernel: ACPI: LAPIC (acpi_id[0x3f] lapic_id[0x3f] disab led)
Nov 19 11:16:33 ptzntwb2 kernel: ACPI: LAPIC (acpi_id[0x40] lapic_id[0x40] disab led)
Nov 19 11:16:33 ptzntwb2 kernel: ACPI: LAPIC (acpi_id[0x41] lapic_id[0x41] disab led)
Nov 19 11:16:33 ptzntwb2 kernel: ACPI: LAPIC (acpi_id[0x42] lapic_id[0x42] disab led)
Nov 19 11:16:33 ptzntwb2 kernel: ACPI: LAPIC (acpi_id[0x43] lapic_id[0x43] disab led)
Nov 19 11:16:33 ptzntwb2 kernel: ACPI: LAPIC (acpi_id[0x44] lapic_id[0x44] disab led)
Nov 19 11:16:33 ptzntwb2 kernel: ACPI: LAPIC (acpi_id[0x45] lapic_id[0x45] disab led)
Nov 19 11:16:33 ptzntwb2 kernel: ACPI: LAPIC (acpi_id[0x46] lapic_id[0x46] disab led)
Nov 19 11:16:33 ptzntwb2 kernel: ACPI: LAPIC (acpi_id[0x47] lapic_id[0x47] disab led)
Nov 19 11:16:33 ptzntwb2 kernel: ACPI: LAPIC (acpi_id[0x48] lapic_id[0x48] disab l
p===============================





Nov 19 08:16:13 ptzntap kernel: e820: [mem 0xc0000000-0xefffffff] available for PCI devices
Nov 19 08:16:13 ptzntap kernel: Booting paravirtualized kernel on bare hardware
Nov 19 08:16:13 ptzntap kernel: setup_percpu: NR_CPUS:4096 nr_cpumask_bits:128 n r_cpu_ids:128 nr_node_ids:1
Nov 19 08:16:13 ptzntap kernel: PERCPU: Embedded 27 pages/cpu @ffff880ffe600000 s81664 r8192 d20736 u131072
Nov 19 08:16:13 ptzntap kernel: Built 1 zonelists in Zone order, mobility groupi ng on. Total pages: 16514904
Nov 19 08:16:13 ptzntap kernel: Policy zone: Normal
Nov 19 08:16:13 ptzntap kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-3.8.13- 55.1.6.el7uek.x86_64 root=/dev/mapper/ol-root ro crashkernel=260M@0M rd.lvm.lv=o l/root rd.lvm.lv=ol/swap rhgb quiet
Nov 19 08:16:13 ptzntap kernel: PID hash table entries: 4096 (order: 3, 32768 by tes)
Nov 19 08:16:13 ptzntap kernel: __ex_table already sorted, skipping sort
Nov 19 08:16:13 ptzntap kernel: xsave: enabled xstate_bv 0x7, cntxt size 0x340
Nov 19 08:16:13 ptzntap kernel: Checking aperture...
Nov 19 08:16:13 ptzntap kernel: No AGP bridge found
Nov 19 08:16:13 ptzntap kernel: Memory: 65679928k/68157440k available (5628k ker nel code, 1049224k absent, 1428288k reserved, 4175k data, 1580k init)
Nov 19 08:16:13 ptzntap kernel: Hierarchical RCU implementation.
Nov 19 08:16:13 ptzntap kernel: RCU restricting CPUs from NR_CPUS=4096 t o nr_cpu_ids=128.
Nov 19 08:16:13 ptzntap kernel: NR_IRQS:262400 nr_irqs:1704 16
Nov 19 08:16:13 ptzntap kernel: Extended CMOS year: 2000
Nov 19 08:16:13 ptzntap kernel: Console: colour VGA+ 80x25
Nov 19 08:16:13 ptzntap kernel: console [tty0] enabled
Nov 19 08:16:13 ptzntap kernel: allocated 268435456 bytes of page_cgroup
Nov 19 08:16:13 ptzntap kernel: please try 'cgroup_disable=memory' option if you
================================


Nov 19 08:16:13 ptzntap kernel: ata3: SATA max UDMA/133 abar m4096@0xfd5ff000 po rt 0xfd5ff100 irq 75
Nov 19 08:16:13 ptzntap kernel: ata4: SATA max UDMA/133 abar m4096@0xfd5ff000 po rt 0xfd5ff180 irq 75
Nov 19 08:16:13 ptzntap kernel: ata5: SATA max UDMA/133 abar m4096@0xfd5ff000 po rt 0xfd5ff200 irq 75
Nov 19 08:16:13 ptzntap kernel: ata6: SATA max UDMA/133 abar m4096@0xfd5ff000 po rt 0xfd5ff280 irq 75
Nov 19 08:16:13 ptzntap kernel: ata7: SATA max UDMA/133 abar m4096@0xfd5ff000 po rt 0xfd5ff300 irq 75
Nov 19 08:16:13 ptzntap kernel: ata8: SATA max UDMA/133 abar m4096@0xfd5ff000 po rt 0xfd5ff380 irq 75
Nov 19 08:16:13 ptzntap kernel: ata9: SATA max UDMA/133 abar m4096@0xfd5ff000 po rt 0xfd5ff400 irq 75
Nov 19 08:16:13 ptzntap kernel: ata10: SATA max UDMA/133 abar m4096@0xfd5ff000 p ort 0xfd5ff480 irq 75
Nov 19 08:16:13 ptzntap kernel: ata11: SATA max UDMA/133 abar m4096@0xfd5ff000 p ort 0xfd5ff500 irq 75

============================
Please help me on this problem am not very well in Linux.


BR
Ralph

From what you wrote it sounds as if you're saying the VM guest itself is staying up and it is only the Oracle processes that "stop". You mention RAC which is Oracle Real Application Cluster which implies this is one node of a cluster.

1) Is it only Oracle processes that "stop" or are you saying the VM guest shuts down completely. (i.e. Are you logging in then doing a reboot or is it refusing logins?)
2) If it is only the Oracle processes do services fail over to another "node"?

I've seen issues where Oracle RAC stopped a node's processes do to tiny network bumps. You'd have to examine the Oracle logs rather than the system logs to see why the processes stopped and/or failed over.

3) For your system log are you showing us what happened after the reboot? If so what is in the log just BEFORE the processes stop?

Since you're running OEL and Oracle RAC you ought to be able to open a ticket with Oracle support so they can analyze what is happening.

1 members found this post helpful.

Hi Mensar,
See my comments in line

From what you wrote it sounds as if you're saying the VM guest itself is staying up and it is only the Oracle processes that "stop". You mention RAC which is Oracle Real Application Cluster which implies this is one node of a cluster. It is not Oracle process stop, i cannot ssh or access the server at all.

1) Is it only Oracle processes that "stop" or are you saying the VM guest shuts down completely. (i.e. Are you logging in then doing a reboot or is it refusing logins?)[Am not so sure if the Oracle stops the server, need more clarifications on this, how Oracle can stop the server not accessible
2) If it is only the Oracle processes do services fail over to another "node"? [I can only access the server through VMware Console, but not directly to the server using ssh]

I've seen issues where Oracle RAC stopped a node's processes do to tiny network bumps. You'd have to examine the Oracle logs rather than the system logs to see why the processes stopped and/or failed over. [Yes we have RAC but still am not so sure if this is because of RAC, Can you assure me that RAC can be installed and configured to the VMware? is possible perhaps i should not using VMware to implement RAC.]

3) For your system log are you showing us what happened after the reboot? If so what is in the log just BEFORE the processes stop? [Am using the VM console to restart the box, then i will be able to login and get logs, but why am not getting the old logs which will show me when the problem started and perhaps caused by what?
Am i looking to the correct logs which will report everything in the server? am looking into messages logs.

Since you're running OEL and Oracle RAC you ought to be able to open a ticket with Oracle support so they can analyze what is happening.[Sure will do so.]
Please help me to analyze this as days going on am getting a lot of system instability hence affect performance of the system.

BR
Ralph

(replies bolded to make it easier to read): So the answer is "The server is staying up, but we can't access it over the network", correct???
You need to read the Oracle documentation on RAC, which will answer your questions, or as you were advised, contact Oracle support. RAC and Oracle DB are NOT FREE, and you pay for support.
Because you're not copying them off before you do a restart??? Since you're looking at them via console, copy/paste/move those files, before restarting the machine, and examine them afterwards.
Since you're going to contact Oracle, then **THEY** can help you analyze things. Again, your system logs will tell you about the system, and if you're the administrator, you should be able to analyze/spot anything suspicious. Post the relevant parts back here and we can try to help, but if you're asking US to do your system diagnosis for you, that's something you need to pay for, since you're (essentially), asking us to do the job of your system administrator. Past that, Oracle can help you with the Oracle database/RAC diagnosis.

And does this happen on a schedule? Same time/day each time? When a particular process runs? What (if any) database housekeeping is done, and does it coincide with the system problems? Have you talked with your DBA's?

You say you can only access via the VMWare console. Once you go there is it allowing login to the OEL operating system or are you saying you can only reboot there? If you are rebooting via VMWare console exactly how are you doing it? (e.g. Are you telling VMWare to stop the guest then restart it or are you able to use OEL operating system GUI or commands to do a true reboot?

If the issue is you can login to the OEL OS via VMWare console but not via ssh it suggests you have a networking issue of some sort preventing access via ssh. Does ping work? If you can't login either way it suggests the issue is server level. There are many things that might cause this including runaway processes.

What version of OEL are you running?

/var/log/messages would give you what is happening at OS level. I was asking you to look at the time stamps on what you're seeing after the boot to see any messages that occurred before you did the reboot. The boot itself puts a lot into messages which you can mostly disregard as they won't be the cause of the hang - messages before the boot may tell you something about the cause.

Oracle RAC and its components (CRS etc...) have their own logs. Oracle support should be able to let you know what to look at.

If you're getting logged

1 members found this post helpful.

Hi TBONe,
Will respond to your questions as follows
So the answer is "The server is staying up, but we can't access it over the network", correct??? Yes it over the network we cant access, rather we log in to the console through Vcenter and power it off and on (restart) as we cant managed to log in to the server (guest) , we can not find logs as we are yet to login to the server until we reboot.

You need to read the Oracle documentation on RAC, which will answer your questions, or as you were advised, contact Oracle support. RAC and Oracle DB are NOT FREE, and you pay for support. [Oracle DBA said there is no problem from the DB, it is OS thus why am looking for linux expert i can pay if need be. send me through my email rligallama at crdbbank.com

Hi Mensar,
See my comments in line below,
ou say you can only access via the VMWare console. Once you go there is it allowing login to the OEL operating system or are you saying you can only reboot there? If you are rebooting via VMWare console exactly how are you doing it? (e.g. Are you telling VMWare to stop the guest then restart it or are you able to use OEL operating system GUI or commands to do a true reboot?
[Yes we can only reboot from console by power off and on, then we can managed to login and ssh]

If the issue is you can login to the OEL OS via VMWare console but not via ssh it suggests you have a networking issue of some sort preventing access via ssh. Does ping work? If you can't login either way it suggests the issue is server level. There are many things that might cause this including runaway processes.[ It is not Network issues rather is a OS (environment ) the server sometimes you PING and not ssh sometimes you can not PING or SSH as well but the server through console is seen until again power it off and on.


What version of OEL are you running? we are running the OEL version 7.1 (Oracle Linux)


/var/log/messages would give you what is happening at OS level. I was asking you to look at the time stamps on what you're seeing after the boot to see any messages that occurred before you did the reboot. The boot itself puts a lot into messages which you can mostly disregard as they won't be the cause of the hang - messages before the boot may tell you something about the cause.
[ON /var/log/messages only i found there is after restart the box which does not showing the root cause for the server to freeze or hang. which logs can i get the previous logs before restart?
We need help we can send me email through rligallama at crdbbank.com

BR
Ralph

So you CANNOT log in to the server, then??? If the only thing you can do is power-cycle it via Vcenter, then the server locks up. Where have you looked for the logs? Because even if you're using journalctl to view the logs, you can go back a good ways. And once the server is back up, you can modify system logging to save things, so not sure where the problem comes in about looking at the logs.
No, thanks. Again, you need to CONTACT ORACLE SUPPORT, period. THEY can help you with doing Oracle diagnosis, and looking at what's actually going on. We aren't going to email you, and I doubt anyone here would just work without getting paid up front. If you want a 'Linux expert', then you should be paying for Oracle Linux support, and just call them...that's what you pay for. You say this is effecting your *CUSTOMERS*, meaning you're running a business. Paying for professional support should be the first thing you do, as your customers are paying YOU for support.

TBONe,
Other servers are not having Oracle, it is just Linux with webservices but still behave the same thing, so i might say it is not Oracle issues at all.


BR
Ralph

..and it MIGHT BE, since you've done basically no diagnostics as of yet. You've not looked at system logs, or had Oracle check RAC. In short, you still have no idea what's going on. And in your first post you specifically said, "Other servers run the Oracle RAC, which also affect the services to my customers.", meaning that there ARE other servers that are part of the RAC. So which is it?? Multiple servers, or just one?

What SPECIFIC issues are you having, since you're now implying that it's "webservices" that are the problem, not just Oracle? And what details do you have? You only say Oracle Linux, but tell us absolutely nothing about what version, version of Oracle DB, RAC, what these other servers are running, what kind of hardware, or VMware version. You cannot expect us to just start guessing, when you've not told us much.

AGAIN:

• Check your system logs; if you do not know where they are, contact your systems administrator. Until you have SOME idea what's going on, you can do nothing.
• Get with Oracle and open a support case, and *KNOW* if you have a problem there. Again, Oracle is not free, nor is RAC. If you want to pay someone here, you should then have no problems paying for support.

which logs apart from var/log/messages is there any logs which records all server activities.?


BR
Ralph

*AGAIN*:

• Get your systems administrator involved. They should know what to look at.
• You provide no details, so we don't know what file(s) things may be going to, if they're off to a central syslog server, or if you have to use journalctl to look at things. We cannot guess.
• You were asked several questions previously; you have not answered ANY of them. Either answer questions and show some effort, or there's little point in posting.
• CALL ORACLE SUPPORT, since you're paying for it.

Can't be more plain about any of this.

That is incorrect.

Since the Operating System (OS) is OEL 7.x it is "Oracle" issues even if it is not "Oracle DB" issues. OEL = Oracle Enterprise Linux. That is Oracle's own distribution based on RHEL (RedHat Enterprise Linux). You would have gotten OEL from Oracle and Oracle would be the organization that supports it. I'd suggest you open a ticket with Oracle for the suspected OS issue.

P.S. Did the DBAs actually examine their logs? Did they see a failover from this node to another one before the boot? If not did they see a failover during the boot? Did they see it fail back after the boot?

Part of the issue with answering questions is use of inexact statements. I asked what you meant in my last post because you suggested you were logging into the console of the OEL guest. Your responses make it clear you're NOT logging into the console of the OEL guest. You're logging into the VMWare hypervisor and telling it to bounce the OEL guest. That is the equivalent of "pushing the button" (power cycling) on a stand alone installation. The fact you have to do that is why I now say your OS "hung".

While it may be the OS "hung" it is typically something doing unreasonable things (e.g. eating all resources thereby preventing other things from getting resources) that cause this kind of thing if it isn't a hardware issue. You could look through sar output for the time period in question to see if any of the output shows steady growth (or shrinkage) before the hang. If you have a tool like atop installed and gathering samples you could look through it to see if any processes it tracked show increases in CPU, memory, disk or network before the boot.

Since this is a VM Guest it isn't likely to be hardware as hardware should also affect other guests on the same hypervisor.

1 members found this post helpful.

@Mensar,
You are very correct, could be the resources in VM environment, can you assist us we will pay you to please to identify the root cause, am sure you can help us a lot.

BR
Ralph

And so could I, and many others on this site. But you, **AGAIN**, do not answer any questions asked. If you're not even going to talk with us or answer questions, are we supposed to just trust you that you'd pay???

And it's VERY hard to believe you'll pay someone here, when you can just call Oracle and open a support ticket. This is your company, you're paying for support...use it. Why are these things hard to understand for you?? I raised the same issue about getting the DBA's involved, calling Oracle, etc. And again, you were asked about the VMware version, because that too, **IS NOT FREE** and you can call THEM for support as well.

Seems little point in you posting...you seem to want someone to just fix your problems for you, with the promise of 'you'll pay'. Personally, your story just isn't adding up,
and you're not communicating (which would make it hard to work with you).