I run Montavista 2.6.18 on ARM/TI.
I need to use external loopback on ttyS0, which is linux console by default.
The application will test the whole serial track via loopback.

It is started from initd (script in rc3.d) and getty start in iniittab is allowed as "once" only (not respawn).
The reduced initialization code looks like:

system("killall getty");
out = open("/dev/null", O_WRONLY);
dup2(out, fileno(stdout))
fd = open("/dev/ttyS0", O_RDWR | O_NOCTTY | O_NONBLOCK);
tcgetattr(fd, &Opts);
cfmakeraw(&Opts);
Opts.c_cflag &= ~CRTSCTS;
Opts.c_cflag |= CLOCAL | CREAD;
tcsetattr(fd,TCSANOW,&Opts);

Now, the questions/problems:
1. When the first line of the initialization code system("killall getty") is active, application hangs at this point (never exits system() call), if the loopback is closed. Otherwise it passes OK, the test starts, and even passes, when I close the loopback later.

2. Removing system() call allows the application to initialize and pass to the first write(fd,"Test text",10), which hangs.
Again the same - starting with loopback opened and closing it later after makes everything working as required.

Any hints/recommendations/help will be highly appreciated...

You have to disable ttys0 from being started for login.

The problem is that the init process will restart the getty process as soon as it exits. The only to way to prevent that is to stop init from starting it. And that requires you to change the configuration for init. Otherwise you are in a race condition between init respawining getty before you get your test started.

I'm not familiar with Montavista, but usually the console terminals are started in /etc/inittab.

For such testing, I would suggest using an unused runlevel that doesn't initialize a getty ttys0 (try to have an alternate though). This runlevel can then be used to run your diagnostics. When they have finished, have the diagnostic script switch run levels to something else.

Thank you, JPollard (Jonathan Pollard? My Respect!

May be you did not noticed that I start getty in inittab with option "once" instead of "respawn" in hope that this will allow one login, if required. Therefore I do kill getty, when the testing starts. Otherwise, one will still be able to login...

But you point to the right way - I tried to remove getty at all and found that after this everything works fine!
SO, the issue is - how to allow getty to run when there is no need in testing (most of the work).

That was why I mentioned using an "unused" run level. Going to that run level would stop the getty from running, when the tests would be done, the last thing the script would do is switch runlevels to the default. To start the test, just tell init to switch to the test run level. The current login would be terminated (getty execs login, login execs shell, so killing the process that was started by getty will terminate the login).