I have an embedded ubuntu 14.04 image and I'm accessing it over ssh and also have apache serving a site.

Sometimes, apparently randomly I get a strange issues in which any page I try to access on apache I get a Forbidden error.

Simultaneously, if I am connected on ssh, I am unable to execute any command, even if I am in 'sudu su' root mode. All commands return 'No such file or directory' even if I specify the full command path. Interestingly a few commands such as pwd and echo do work, others such as ls, reboot, cat, ping, ...all give that error.

If I am not connected to ssh, trying to connect to ssh doesn't work.

It seems as if all of a sudden there is a general loss of permissions. I'm not sure if it eventually recovers automatically as I always end up doing a power cycle.

I know this is quite generic but that is all I know. syslog doesn;t seem to indicate anything abnormal.

pwd and echo are built into the shell, whereas cat, ls, ping and so on are executables residing on files. That fact could be the start of a clue. Something wrong with the root directory. Or the Loader, typically residing on /lib/ldblabla.so or perhaps /lib64, is inaccessible. Or the loader's config, ld.so.conf and ld.so.cache, corrupted.

It might help if you could provide:

• precise commands and error messages
• context: when it happens, does it go away by it's own? Do you need to log out? Reboot?
• do you log on, work normally, then suddenly commands don't work?

Hi Bernd, thanks for your reply.

To put you in the picture this is a custom designed i.MX53 board running a 32bit Ubuntu 14.04.1 LTS image and 3.16.2-armv7 kernel. The board has LAN interface and also a pppd connection over cell modem. I am accessing it using an ssh connection over the LAN interface. The board is constantly running in a semi-remote location and I only have network (LAN interface) access to it.

So basically the context in which this happens is that I have this running 24/7 and sometimes (to me it appears randomly) I loose access to it. Seems to be happening every couple of weeks but sometimes more often. I can ping it fine and I can tell some custom processes are still running as I can see it poll an external modbus device but trying to access any page via the webserver I get a Forbidden error:


I sometime leave an ssh connection active for a few days, in that case when the issues happens - as I said previously - all commands return a 'No such file or directory' . that is the only error message I get to any command I try. Good point on pwd and echo are built into the shell. Will check the loader's cache maybe it is being corrupted as you say.

If I happen to not be connected to ssh when this happens, I simply cannot access ssh, failing with this error:

To get the system back into normal operation I need to get someone to power cycle it. Until someone can go to power cycle it sometimes it takes a few days and it has never recovered during this time except once it seemed to have recovered by itself after 2 days (and uptime was showing it had not rebooted). I'm not sure if in that particular case there was some other factor, in general it doesn't seem to recover by itself.

Cannot really figure out a sequence of events that leads to the issue. At the moment I have it in this condition but unfortunately I have just now lost ssh connection to it (...was connected via teamviewer to a windows PC on the remote network and ssh to the board from there and stupid windows rebooted to install updates!!!)

I have a suspicion (although not a solid one) that the pppd connection has something to do with this. Meanwhile I have the 'same' image running on an identical local board and identical setup and I never managed to replicate the issue. The images are not exactly identical but both started off from the same image and have the same changes from there onwards. Also I don't have any indication that it is hardware related.

After the power cycle it works perfectly normal, and works for several days without any problem. I can login to ssh and everything is normal then it just happens all of a sudden, normally it happens while I am not using it so I leave it working fine and the next day i find ssh still connected but unusable, or I can no longer connect to ssh.

Thanks!

That's an interesting troubleshooting challenge. I wonder what happens to your ssh connection. Does it close the connection because it can't spawn a shell?

A few general ideas, most related to logging.
When you can't connect, use ssh -vvv (or is it -ddd?) - that produces an ocean of debug messages and perhaps a clue what's wrong with the ssh server.
On the server side, you can run a second (or third, ...) sshd daemon in parallel, listening to a different port, with a -vvv option and collecting its messages in a log file. When you don't get connected, will all sshd's refuse to connect?
How about cranking up the Apache log level.
I suppose you have checked if there is a time correlation between all these problems. I see three problem domains - commands not working, unable to connect, web server.

ssh -vvv log:

With regards to time correlation, all 3 issues happen at the same time, I am sure of that.

Well at least we can see some negotiation going on between client and server. I am pretty certain the sshd gives up when it can't exec a program like the shell. Perhaps the sshd's error message will provide some info.

yes I think sshd is unable to exec just like I am unable to run commands when I happen to be connected to ssh. Both apahce and sshd are still running. apache even serves the ssl certificate properly and produces the Forbidden error message. the response time is also very fast so there isn't some process hogging up the resources. I don't think the issue is related to sshd or apache, these are just side-effects. It would seem to be related to some form of random corruption, maybe memory corruption, however I have no indication of memory hardware issues and the system was pretty much idle in the past couple of days so there wasn't much going on. The system has 2GB RAM and at the moment it was only using around 128MB with little variation. There are also no other issues indicating memory problems even when the system is loaded.

I'll check apache log when I get access to it and post some snippets. It should log the reason why it forbids access.

managed to have the board power cycled, here are the interesting points:

apache2 error.log just before the issue occurred (I think):

not sure why those ppp0 lines are there.

access.log has no entry for the duration of the issue.

The last activity in syslog before the issue occured is also related to pppd but everything seems normal:

There is no other entry in syslog until the system is power cycled.

Also nothing interesting in auth.log

I am getting more convinced it is some how related to the pppd connection. The device has become more unstable locking up in this state more often. I did a test in which I disabled pppd connection and so far it hasn't produced the issue. Cannot imaging how the pppd connection can create such an issue though...

digging deeper into the issue, it seems that only when i set 'replacedefaultroute' option in pppd I get this strange lock up issue. So it might simply be a related to a packet routing issue. Still even in this mode LAN traffic coming from the subnet should be routed properly and it seems this issue only happens at some point during normal operation.