LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 08-23-2021, 01:43 PM   #1
Bad Blue Bull
Member
 
Registered: Jul 2017
Distribution: Slackware 14.2
Posts: 62

Rep: Reputation: Disabled
Unbuffered read from stdin in python


I wanna convert telnet output from CP866 to UTF-8, I'm sure it's possible to do by redirecting output because telnet|cat is fine and also redirecting telnet output to file and reading the file with tail -f.
This is the script I tried:
Code:
#!/usr/bin/env python

import io
import sys

input_stream = io.TextIOWrapper(
    sys.stdin.buffer, encoding='cp866')

for line in input_stream:
    print(line)
same result with this:
Code:
#!/usr/bin/env python

import io
import sys

input_stream = io.TextIOWrapper(
    sys.stdin.buffer, encoding='cp866', newline='\n')
while True:
    x = input_stream.read(1)
    if len(x) == 0:
        break
    print(x, end='')
Save it as from866, make it executable and try telnet <target> | from866.
It works fine when output of ls for example is redirected into it, but with telnet it prints all well until login prompt, but then the later stuff is terribly buffered. Login prompt appears only I input login. I looked at gnu cat source, don't see how it manages to get unbuffered input.
 
Old 08-23-2021, 03:14 PM   #2
astrogeek
Moderator
 
Registered: Oct 2008
Distribution: Slackware [64]-X.{0|1|2|37|-current} ::12<=X<=14, FreeBSD_12{.0|.1}
Posts: 5,717
Blog Entries: 11

Rep: Reputation: 3748Reputation: 3748Reputation: 3748Reputation: 3748Reputation: 3748Reputation: 3748Reputation: 3748Reputation: 3748Reputation: 3748Reputation: 3748Reputation: 3748
If I understand your question, piping output through iconv should do what you want.

See man iconv for details.
 
1 members found this post helpful.
Old 08-23-2021, 03:17 PM   #3
Bad Blue Bull
Member
 
Registered: Jul 2017
Distribution: Slackware 14.2
Posts: 62

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by astrogeek View Post
See man iconv for details.
No, iconv waits for EOF, I need it to work like tail -f.
 
Old 08-23-2021, 03:18 PM   #4
Bad Blue Bull
Member
 
Registered: Jul 2017
Distribution: Slackware 14.2
Posts: 62

Original Poster
Rep: Reputation: Disabled
OK, maybe I need to try to write it in C then.
 
Old 08-23-2021, 04:29 PM   #5
boughtonp
Senior Member
 
Registered: Feb 2007
Location: UK
Distribution: Debian
Posts: 1,631

Rep: Reputation: 1330Reputation: 1330Reputation: 1330Reputation: 1330Reputation: 1330Reputation: 1330Reputation: 1330Reputation: 1330Reputation: 1330Reputation: 1330

Searching for "python tail" revealed "PythonTail: a Unix tail implementation in python."

It claims support for "-f, --follow", so perhaps they've already solved the buffering issue you're facing.

 
Old 08-23-2021, 04:35 PM   #6
teckk
Senior Member
 
Registered: Oct 2004
Distribution: FreeBSD Arch
Posts: 3,621
Blog Entries: 2

Rep: Reputation: 1163Reputation: 1163Reputation: 1163Reputation: 1163Reputation: 1163Reputation: 1163Reputation: 1163Reputation: 1163Reputation: 1163
Code:
>>> mytext1 = "Mary had a little lamb"
>>> mytext2 = "Пьищ гьа ь всшшве вьпфл"
>>> b = bytes(mytext1,"CP866") 
>>> print(b)
b'Mary had a little lamb'
>>> c = bytes(mytext1, "UTF-8")
>>> print(c)
b'Mary had a little lamb'
>>> d = bytes(mytext2,"CP866")
>>> print(d)
b'\x8f\xec\xa8\xe9 \xa3\xec\xa0 \xec \xa2\xe1\xe8\xe8\xa2\xa5 \xa2\xec\xaf\xe4\xab'
>>> e = bytes(mytext2, "UTF-8")
>>> print(e)
b'\xd0\x9f\xd1\x8c\xd0\xb8\xd1\x89 \xd0\xb3\xd1\x8c\xd0\xb0 \xd1\x8c \xd0\xb2\xd1\x81\xd1\x88\xd1\x88\xd0\xb2\xd0\xb5 \xd0\xb2\xd1\x8c\xd0\xbf\xd1\x84\xd0\xbb'
>>> print(e.decode())
Пьищ гьа ь всшшве вьпфл
 
Old 08-23-2021, 05:50 PM   #7
Bad Blue Bull
Member
 
Registered: Jul 2017
Distribution: Slackware 14.2
Posts: 62

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by boughtonp View Post
Searching for "python tail" revealed "PythonTail: a Unix tail implementation in python."

It claims support for "-f, --follow", so perhaps they've already solved the buffering issue you're facing.

OK, with little tweaking it can convert file from cp866. Now I need to learn how to read from stdin anyway, because it's an ugly way to use file for output.
There's something fundamental I don't understand. telnet|cat is fine. But telnet|cat|from866 is just as lame as telnet|from866.
 
Old 08-23-2021, 07:37 PM   #8
astrogeek
Moderator
 
Registered: Oct 2008
Distribution: Slackware [64]-X.{0|1|2|37|-current} ::12<=X<=14, FreeBSD_12{.0|.1}
Posts: 5,717
Blog Entries: 11

Rep: Reputation: 3748Reputation: 3748Reputation: 3748Reputation: 3748Reputation: 3748Reputation: 3748Reputation: 3748Reputation: 3748Reputation: 3748Reputation: 3748Reputation: 3748
I am confused. Perhaps if you could describe what problem you are actually trying to solve rather than focusing on the text encodings.

Are you trying to write telnet output to the file, or read from the file as input to telnet, or both? Are you simply trying to capture the telnet session to a file or initiate a telnet session from file contents, and to what end?

Help us understand.
 
Old 08-23-2021, 07:48 PM   #9
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Debian, Arch
Posts: 3,612

Rep: Reputation: 1975Reputation: 1975Reputation: 1975Reputation: 1975Reputation: 1975Reputation: 1975Reputation: 1975Reputation: 1975Reputation: 1975Reputation: 1975Reputation: 1975
Have you tried python's -u argument?

https://manned.org/python.1
Code:
       -u     Force the stdout and stderr streams to be unbuffered.  This option
              has no effect on the stdin stream.
 
Old 08-23-2021, 07:56 PM   #10
Bad Blue Bull
Member
 
Registered: Jul 2017
Distribution: Slackware 14.2
Posts: 62

Original Poster
Rep: Reputation: Disabled
astrogeek: convert telnet output from one encoding to another. telnet still needs to be interactive, iconv can't accomplish it because it can't convert text interactively, it only converts all text from stdin when program that feeds it finishes.
 
Old 08-23-2021, 07:59 PM   #11
Bad Blue Bull
Member
 
Registered: Jul 2017
Distribution: Slackware 14.2
Posts: 62

Original Poster
Rep: Reputation: Disabled
ntubski
Quote:
This option has no effect on the stdin stream.
What I need is unbuffered stdin stream.
Many tools have -u key, but not telnet
 
Old 08-23-2021, 11:17 PM   #12
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 4,206
Blog Entries: 1

Rep: Reputation: 1565Reputation: 1565Reputation: 1565Reputation: 1565Reputation: 1565Reputation: 1565Reputation: 1565Reputation: 1565Reputation: 1565Reputation: 1565Reputation: 1565
How about setting LC_CTYPE to language_country.UTF-8 within the telnet session?

Or, you could use PuTTY's 'translation' functionality.

Last edited by NevemTeve; 08-24-2021 at 02:20 AM.
 
2 members found this post helpful.
Old 08-24-2021, 12:06 PM   #13
Bad Blue Bull
Member
 
Registered: Jul 2017
Distribution: Slackware 14.2
Posts: 62

Original Poster
Rep: Reputation: Disabled
NevemTeve: thanks, I didn't know that PuTTY can do it.
I also thought about changing locale, there are instructions, just too many actions and I wanted more simple solution.
 
Old 08-24-2021, 12:14 PM   #14
Bad Blue Bull
Member
 
Registered: Jul 2017
Distribution: Slackware 14.2
Posts: 62

Original Poster
Rep: Reputation: Disabled
OK, PuTTY serves my purpose, but I think I shouldn't mark the thread as solved because buffered input from stdin is a wider problem and somebody can face it in different situation. Maybe it's more Python related than system related.
 
Old 08-24-2021, 03:11 PM   #15
shruggy
Senior Member
 
Registered: Mar 2020
Posts: 2,552

Rep: Reputation: Disabled
Quote:
Originally Posted by Bad Blue Bull View Post
I also thought about changing locale, there are instructions, just too many actions and I wanted more simple solution.
Then how about luit?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Python can't read script from stdin when input() appears in the script. mimorek Programming 10 04-24-2021 08:14 AM
Sometimes, sed --unbuffered is not unbuffered kubuntu-man Programming 1 04-08-2016 12:16 PM
Python: Stdin and python apps fighting? enigma_0Z Programming 2 07-15-2008 08:45 AM
unbuffered stdin Mike Davies Linux - Software 1 03-03-2006 12:13 AM
unbuffered stdin mvt Programming 7 05-17-2004 09:40 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 03:22 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration