[SOLVED] python output using "python script > result" shows that the sequence of printing is haphazard

centguy · 03-25-2022, 05:30 AM

I have a python script "script.py".

When I run "python script.py" the output on the screen seems to be in sequence.

But if I direct the output to a file via
"python script.py > a-file", the sequence is definitely messed up.

How to preserve the sequence to allow debugging?

The printings from script.py are from print("..."), os.system("cat a-list"),
and os.system("a-Fortran-binary"), where a-Fortran-binary printing output to the screen.

Python is giving me so many surprises..

pan64 · 03-25-2022, 05:32 AM

would be nice to see some details or example.
Do you mean the output of the two commands are mixed?

centguy · 03-25-2022, 10:10 AM

First write a simple Fortran program
:

Quote:

$ cat fort.f90
program simple

write(*,*) 'I am in Fortran!'

end program

Next prepare a simple test.py:

PHP Code:



$ cat test.py
import os


L = [1,2,3]

print("L is ",L)

with open("Lfile","w") as f:

       f.write("L is %d %d %d\n"%(L[0],L[1],L[2]))

os.system("cat Lfile")

os.system("./fortbin")

Then prepare a Makefile:

PHP Code:



$ cat Makefile 
all:
    gfortran fort.f90 -o fortbin
    python test.py
    echo "----separator----"
    python test.py > savepy-output
    cat savepy-output

Now type make:

Quote:

$ make
gfortran fort.f90 -o fortbin
python test.py
L is [1, 2, 3]
L is 1 2 3
I am in Fortran!
echo "----separator----"
----separator----
python test.py > savepy-output
cat savepy-output
L is 1 2 3
I am in Fortran!
L is [1, 2, 3]

See that `L is [1, 2, 3]' appears on the first line in first instance and
the third line in the second instance.

Probably python is not designed to do this kind of things.

boughtonp · 03-25-2022, 11:16 AM

That's not a great example - could have been simplified further to highlight the issue.

Anyway, the docs are a bit vague, but it appears to be because os.system is asynchronous, and you need an explicit os.wait() call to change that.

Or you could replace it with subprocess.call, which will: "Run the command described by args. Wait for command to complete, then return the returncode attribute."

centguy · 03-25-2022, 10:55 PM

>That's not a great example - could have been simplified further to highlight the issue.

i have used all I know to cook up this example.

centguy · 03-25-2022, 11:23 PM

I find it useful to flush the python output first
using

sys.stdout.flush()

A bit clumsy but it works.

centguy · 03-25-2022, 11:38 PM

Actually there are better options:

1. add flush=True so that we have print(any_string,flush=True)

2. Option 1 gets a bit tiring since we have to put flush=True in every single print statement.
To avoid it, ask python not to buffer using "-u" option.

So we need to do "python -u script.py"

option2 seems better since it involves the least of work but we have to remember to use -u.

boughtonp · 03-26-2022, 09:18 AM

As per "python --help" instead of passing "-u" every time you can set an environment variable "PYTHONUNBUFFERED=x"; put it in a startup file and you don't need to remember it.

Not sure this is related to your question though, unless you're saying that preventing buffering has the effect of enforcing atomicity in the os.system calls?

centguy · 03-27-2022, 03:53 AM

PYTHONUNBUFFERED=x

is a good idea. Thanks!

pan64 · 03-28-2022, 12:42 AM

Quote:

Originally Posted by centguy

Actually there are better options:

1. add flush=True so that we have print(any_string,flush=True)

2. Option 1 gets a bit tiring since we have to put flush=True in every single print statement.
To avoid it, ask python not to buffer using "-u" option.

So we need to do "python -u script.py"

option2 seems better since it involves the least of work but we have to remember to use -u.

In case of asynchronous calls non of these options are correct: the order of the execution is unpredictable. Adding flush=True or -u will not change it at all.
But os.system itself is not an async call, it will use wait for the end of the execution, so only these buffers caused this issue (that's why it helped in this case).