LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Hardware > Linux - Embedded & Single-board computer
User Name
Password
Linux - Embedded & Single-board computer This forum is for the discussion of Linux on both embedded devices and single-board computers (such as the Raspberry Pi, BeagleBoard and PandaBoard). Discussions involving Arduino, plug computers and other micro-controller like devices are also welcome.

Notices


Reply
  Search this Thread
Old 08-05-2017, 10:03 AM   #1
grgsaliba
LQ Newbie
 
Registered: Jan 2013
Posts: 14

Rep: Reputation: Disabled
Question unable to execuue commands after ppp connection


I am working on a custom embedded linux board, with the design loosely based on the i.MX53 quickstart-r, this custom hardware also includes a cell modem.

Now, this problem has been haunting me for quite a long time and I cannot figure out what it is and even how to troubleshoot it properly because it is difficult to get any logs when it happens. Essentially, I have LAN and Cellular data connectivity. As long as the cell data connection is disconnected, there is no issue at all, system is 100% stable. When I connect to the cell data (bringing on ppp0), the data connection itself works fine but sometimes (seemingly random) something (I'm not sure what) happens, but when it happens the symptoms are the following:

1. Anything that is running and does not depend on calling external commands or using external files, keeps running. For example if I am connected to the device over SSH, the connection stays active, I also have apache running and it is still (partly) accessible, and I can tell also that other custom software keeps running too.

2. Trying to access or run any external file returns a "No such file or directoy" error. So for example if I happen to be connected to SSH and try to run simple commands such as ls I get: "-bash: /bin/ls: No such file or directory", any command I try to execute returns that error apart from the built bash commands such as cd which work normally. Similarly apache cannot execute PHP code and all other running software is unable to execute commands they depend on.

3. Rebooting the system (manually or via watchdog) restores everything to normal operation.

Now, recently I regenerated the linux image from scratch with newer kernel, newer ubuntu, and was hoping that the issue would go away but alas it is still happening. I'm not sure if this could be related to hardware in any way but as I said the hardware is working pretty solid (and has been for a couple of years and in numerous units produced).

My main issue is that I seems to be completely unable to troubleshoot this or maybe log enough details to understand what is going on, I tried to look for a pattern or where syslog stop logging but it doesn't lead anywhere.

I know this is somewhat vague but any idea would really help
 
Old 08-07-2017, 04:35 AM   #2
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 6,971
Blog Entries: 4

Rep: Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669
Quote:
Originally Posted by grgsaliba View Post
Trying to access or run any external file returns a "No such file or directoy" error. So for example if I happen to be connected to SSH and try to run simple commands such as ls I get: "-bash: /bin/ls: No such file or directory", any command I try to execute returns that error apart from the built bash commands such as cd which work normally. Similarly apache cannot execute PHP code and all other running software is unable to execute commands they depend on.
maybe the ppp connection messes up your partitioning and drive numbers get shifted around? like, the ppp modem becomes /dev/sda1, and /dev/sda1 (with the whole linux install) becomes /dev/sda2, or /dev/sdb1, or unmounted completely?
something like that (i'm mostly guessing, and the terminology is probably off).
usually the solution is to use UUIDs.

Quote:
I know this is somewhat vague
yes it is.
much more general info please, first troubleshooting steps etc...
 
Old 08-07-2017, 03:08 PM   #3
grgsaliba
LQ Newbie
 
Registered: Jan 2013
Posts: 14

Original Poster
Rep: Reputation: Disabled
I can still move through directories using 'cd' command (this is a builtin bash command so keep working fine) so I'm assuming the root filesystem stays mounted and accessible as such.
 
Old 08-18-2017, 03:59 AM   #4
grgsaliba
LQ Newbie
 
Registered: Jan 2013
Posts: 14

Original Poster
Rep: Reputation: Disabled
just stumbled across an interesting fact, maybe the solution.
Earlier I was using pppd directly, with chat scripts, etc... and this problem was happening randomly. Now since I have regenerated the full system image I also changed to connman to handle all networking - but still I was getting the same issue and it actually seemed to become more frequent.

I've been searching around and noticed some complaints that ppp can cause kernel panic, so I decided to try to uninstall ppp from my system (wasn't sure if connman needed it) and so far after uninstalling ppp I am still able to connect to the cell network via connman but the issue is no longer happening. Since it is a random issue it is too early to say for sure but so far tested on two systems and the issue has not happened yet!
 
Old 08-19-2017, 11:49 PM   #5
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 6,971
Blog Entries: 4

Rep: Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669
mobile broadband + ssh...

maybe you should look at mosh.
 
Old 08-20-2017, 03:07 AM   #6
grgsaliba
LQ Newbie
 
Registered: Jan 2013
Posts: 14

Original Poster
Rep: Reputation: Disabled
I'll check this out but the problem is not as such ssh. ssh is not used at all in the application (only for troubleshooting). The problem is that just like commands stop working over ssh they also stop working internally. So the system becomes dysfunctional as some processes keep running but are no longer able to execute commands.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
etablish one connection ppp between a client ppp and a server ppp camilla261086 Linux - Server 2 07-20-2010 08:09 AM
ppp connection camilla261086 Linux - Networking 1 07-16-2010 12:26 PM
PPP Chat - optional / non-critical commands Critcho Linux - Software 1 10-22-2006 10:57 PM
ppp connection aps_rewa Linux - Networking 0 05-20-2004 05:39 AM
ppp connection help LUXX Linux - Newbie 4 11-14-2003 02:06 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Hardware > Linux - Embedded & Single-board computer

All times are GMT -5. The time now is 07:16 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration