LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 08-27-2016, 01:51 AM   #1
Michael Uplawski
Member
 
Registered: Dec 2015
Location: Normandy, France
Distribution: Debian buster/sid
Posts: 539
Blog Entries: 18

Rep: Reputation: 353Reputation: 353Reputation: 353Reputation: 353
[Ruby] Cannot read from STDIN after piping-in a file


Good morning.

I need a hint on reading from STDIN and the IOCTL options. This may or may not be Ruby-specific. Sorry for the long text. The whole code is available, selected code-sections are quoted below.

Context: I wrote a small Ruby-program to visualize multi-spreadsheet
workbooks in a terminal. Deriving the mime-type of the spreadsheet content
allows me to choose a wrapper-instance adapted for ODS, XLS or XLSX-files. The
external module (“Ruby-Gem” Roo) which does most of the work, however
insists on the file-extension to correspond to the type of the chosen
wrapper
.., meaning: I know from the mime-type, that I need an ODS-wrapper,
but when my file does not have an *.ods extension, an exception is raised and
that's it.

This is normally not a problem. Calls like the following
Code:
user@machine:~/[path]/viewworkbook_trunk/test$ ../bin/viewworkbook.rb ./some_spreadsheet_file.xls
produce output like in the screen shot, below.

Edit: The issue with the file-extension is just an inconvenience
and *not* the core problem that I seek help for in this discussion. It
does, though, impose code which interferes with the function that pipes-in
data from STDIN.

In the screen shot, PSE note the line of options,
below the table. The program is interactive, in that it lets the user move
through the sheets in the workbook, by name or number, adapt the width of the
table-columns or save a single table or the whole workbook to a new file.

Reproducing the error: This program shall be used in a macro to quickly open
email-attachments from Mutt, my email-client. For the macro, I must be able to
pipe-in the file-content to the Ruby-program. As the wrapper class only works
with files of the right filename-extension, I must first redirect the
spreadsheet content to a temporary file, give it the right extension, then
open it just like any other spreadsheet-file.

Here my program crashes after the generation of the table. in the terminal, I
see the complete, first spreadsheet from the workbook and the user options
(like in the screen shot). The following exception is raised when the program
starts to wait for user-input:

Code:
/[path]/viewworkbook_trunk/lib/user_input.rb:30:in `raw': Inappropriate ioctl for device (Errno::ENOTTY)
	from /[path]/viewworkbook_trunk/lib/user_input.rb:30:in `wait_for_user'
	from /[path]/viewworkbook_trunk/lib/menu.rb:80:in `show'
        [...]
I show you some of what I consider relevant code because it reads from STDIN.
As the difference in the working and the failing calls to my program is in the
way, that spreadsheet content is read, there is clearly something missing in
my handling the standard input device, and probably something which is
important in any such routine, independent of the programming language used.

Code
Here is the code from the main class, which “pipes in” spreadsheet content:
Code:
	# verify that the programm argument is '-' 
	if args.length == 1 && args[0] == '-'
			# not all the code is really needed for 
			# the current function to work. But I leave it here
			# for authenicity.
			args.compact!
			args.shift
			# First, I verified, that STDIN can be read.
			# @log.debug('file piped-in: ' << $<.read)
			tfile = nil
			# Verify that there is data on STDIN
			if(!$<.eof?)	
				@log.debug('reading from STDIN')
				# ... provide a temporary file in that case,
				tfile = Tempfile.new('spreadsheet_temp')
				File.open(tfile, 'w+') {|of| of << $<.read}
				@log.debug("Temporary file %s is written." %File.path(tfile))
				
				wb = tfile
			end
		end
After this, the workbook is entirely stored in the file ‘wb’ which still has
no file-extension, but is named something like
“/tmp/spreadsheet_temp20160827-5802-1s6jlg1”.

For the recognition of spreadsheet types, I defined the Mime-types for use
with the magic library, via the Ruby Gem “FileMagic”.
Code:
ODS_MIME = "application/vnd.oasis.opendocument.spreadsheet"
XLS_MIME = "application/vnd.ms-excel"
XLSX_MIME = "application/zip"
CSV_MIME = "text/plain"
The wrapper is generated and the file-extension is added to the temporary
file, if needed in a (“static”) factory class.
Code:
	def self::workbook(file)
		@@file = file
		if !@@workbook
			begin
				@@workbook = case mime_type
					     when ODS_MIME 
						     @@log.debug('ODS')
						     unless File.extname(@@file) == '.ods'
							     odsfile = @@file.dup << '.ods'
							     File.rename(@@file, odsfile) 
							     @@file = odsfile
						     end
						     # create the OpenOffice instance
						     Roo::OpenOffice.new(File.path(@@file)
					      when XLS_MIME
						(...)
					      when XLSX_MIME
						(...)
Then, the menu is created below the table, and we wait for user input, when
the program crashes:
Code:
require 'io/wait'
require 'io/console'

def wait_for_user()
	@log.debug('wait_for_user() ')
        char = STDIN.raw(&:getc)	

# Here is another way to code the non-blocking 
# read from STDIN which usually works, but fails in
# the exact same way after a spreadsheet has been
# piped-in 

=begin
	char = nil
	STDIN.raw do 
		STDIN.noecho do
			until (STDIN.ready?)
				sleep(0.1)
			end

			char = (STDIN.read_nonblock(1).ord rescue nil)
		end
	end
=end
	return char
end
The whole program in its current development-version 0.0.5 is available as a Ruby-gem,
containing all the code, from rubygem.org, either from the site:
https://rubygems.org/gems/viewworkbook
or by installing it with the gem-tool:
Code:
gem install viewworkbook
This is enough to see the complete code. To run the program, you also need the
Roo and the FileMagic gems.
Attached Thumbnails
Click image for larger version

Name:	sc_viewworkbook.jpg
Views:	19
Size:	107.9 KB
ID:	22869  

Last edited by Michael Uplawski; 08-27-2016 at 04:44 AM. Reason: Mostly bad English and cosmetics. WIP...
 
Old 08-27-2016, 07:14 AM   #2
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,552

Rep: Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899
I am still having a bit of a look at things, but may I ask what version of ruby are you using? I have seen a number of these types of errors related to 1.9.3, so it may be nothing but thought I should ask before going too much further
 
1 members found this post helpful.
Old 08-27-2016, 08:21 AM   #3
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Debian, Arch
Posts: 3,241

Rep: Reputation: 1407Reputation: 1407Reputation: 1407Reputation: 1407Reputation: 1407Reputation: 1407Reputation: 1407Reputation: 1407Reputation: 1407Reputation: 1407
Quote:
Originally Posted by Michael Uplawski View Post
This is normally not a problem. Calls like the following
Code:
user@machine:~/[path]/viewworkbook_trunk/test$ ../bin/viewworkbook.rb ./some_spreadsheet_file.xls
produce output like in the screen shot, below.

Edit: The issue with the file-extension is just an inconvenience
and *not* the core problem that I seek help for in this discussion. It
does, though, impose code which interferes with the function that pipes-in
data from STDIN.
So, is the failing case equivalent to something like this?
Code:
user@machine:~/[path]/viewworkbook_trunk/test$ ../bin/viewworkbook.rb < ./some_spreadsheet_file.xls
In which case the problem is essentially that you can't read user input from stdin if stdin is already being used as a file pipe.
 
1 members found this post helpful.
Old 08-27-2016, 01:05 PM   #4
Michael Uplawski
Member
 
Registered: Dec 2015
Location: Normandy, France
Distribution: Debian buster/sid
Posts: 539
Blog Entries: 18

Original Poster
Rep: Reputation: 353Reputation: 353Reputation: 353Reputation: 353
Quote:
Originally Posted by grail View Post
I am still having a bit of a look at things, but may I ask what version of ruby are you using? I have seen a number of these types of errors related to 1.9.3, so it may be nothing but thought I should ask before going too much further
Thank you for the idea. Even if I use ruby 2.4.0dev (from the trunk, build on 21/8), I shall install a somewhat “stable” version just to exclude dumb coincidences... ;-) (What's “unstable” anyway, and should I put a trema on coïncidences?)

(Joking)

Michael
 
Old 08-27-2016, 01:12 PM   #5
Michael Uplawski
Member
 
Registered: Dec 2015
Location: Normandy, France
Distribution: Debian buster/sid
Posts: 539
Blog Entries: 18

Original Poster
Rep: Reputation: 353Reputation: 353Reputation: 353Reputation: 353
Quote:
Originally Posted by ntubski View Post
So, is the failing case equivalent to something like this?
Code:
user@machine:~/[path]/viewworkbook_trunk/test$ ../bin/viewworkbook.rb < ./some_spreadsheet_file.xls
In which case the problem is essentially that you can't read user input from stdin if stdin is already being used as a file pipe.
In deed, I deem my cat spreadsheet.(xls|xlsx|ods) | viewworkbook.rb - equivalent to your alternative command line.
However, are you telling me, that there cannot be a program which continues listening on STDIN after ... WAIT!! I see: Do I understand correctly, that the pipe is opened before the execution of my program and there is just nothing that I can do do close it “during” the execution of the program? ... That would be an occasion to pay a visit to the Calva distillery in the neighborhood.
 
Old 08-27-2016, 01:56 PM   #6
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,552

Rep: Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899
I think your summation is correct but will let ntubski confirm.

I had 2 questions though:

1. Why would you prefer the piped / redirected version over simply calling your program with the file?

2. If you took out the redirection process wouldn't it be simpler to test the file and rename accordingly (temp file if need be) on the actual file and the open it inside your ruby code? (more curious than anything)
 
Old 08-27-2016, 02:18 PM   #7
Michael Uplawski
Member
 
Registered: Dec 2015
Location: Normandy, France
Distribution: Debian buster/sid
Posts: 539
Blog Entries: 18

Original Poster
Rep: Reputation: 353Reputation: 353Reputation: 353Reputation: 353
Quote:
Originally Posted by grail View Post
1. Why would you prefer the piped / redirected version over simply calling your program with the file?
This redirection would be needed to pipe in attachments from my mail-client (Mutt). A macro command can be defined, like one of those, that I use already:
Code:
macro index M "|maillinks -\n"          "create link-list"
macro pager M "|maillinks -\n"          "create link-list"
This pipes a html-mail to another program “maillinks” which shows me the URLs of included html-links in a numbered list. I imagine a similar macro to view spreadsheets that I receive as mail-attachment:
Code:
macro attach <F12> "|viewworkbook.rb -\n" "show spreadsheet"
Without the pipe, I would have to save the attachment first to a file and it would make the call to viewworkbook superfluous as I can run a full-blown spreadsheet program on the file right away...

Quote:
2. If you took out the redirection process wouldn't it be simpler to test the file and rename accordingly (temp file if need be) on the actual file and the open it inside your ruby code? (more curious than anything)
When I do not use the pipe but call the program with a spreadsheet as argument, no renaming is necessary, as long as the files have the correct file-extension.

Thanks for asking. I reconsider all my decisions, each time that a question is formulated in a different way than mine.

Just for completeness, there is a second use for this programming exercise. Versions of the SoftMaker Office Suite before 2015 stored spreadsheets in a format PMD which identifies as “Composite Document File V2 Document, Little Endian”, i.e. the same as XLS. As this office suite does not come with a scripting interface, I hope to manipulate these files anyway from outside the spreadsheet program. There is no connection between this aspect and the pipe-trouble. Even from inside the SoftMaker program, I cannot call external programs on a current file but must open it in a separate process.

Last edited by Michael Uplawski; 08-27-2016 at 02:19 PM.
 
1 members found this post helpful.
Old 08-27-2016, 03:58 PM   #8
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Debian, Arch
Posts: 3,241

Rep: Reputation: 1407Reputation: 1407Reputation: 1407Reputation: 1407Reputation: 1407Reputation: 1407Reputation: 1407Reputation: 1407Reputation: 1407Reputation: 1407
Quote:
Originally Posted by Michael Uplawski View Post
In deed, I deem my cat spreadsheet.(xls|xlsx|ods) | viewworkbook.rb - equivalent to your alternative command line.
Yup.

Quote:
However, are you telling me, that there cannot be a program which continues listening on STDIN after ... WAIT!! I see: Do I understand correctly, that the pipe is opened before the execution of my program and there is just nothing that I can do do close it “during” the execution of the program?
You can close it, but then you just have a closed pipe. Nothing you can do with that. The real problem is that you never get the file handle of the terminal that would let you talk to the user. Maybe you can try opening "/dev/tty"?
 
1 members found this post helpful.
Old 08-27-2016, 04:17 PM   #9
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,552

Rep: Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899
Thanks for all the information ... I wasn't aware just how much mutt could do

Your information did give me one side thought though. When you come across a file with the incorrect file-extension, you then go through the process of renaming this to make the process work.
Which, if I understand correctly, will then involve creating a temp file on your system. On the small reading I have done on mutt you could implement the same idea for your macro by copying the attachment
to /tmp and then performing any necessary tasks, including calling you ruby program on the file. Just a thought as this would solve the pipe dilemma currently vexing you
 
1 members found this post helpful.
Old 08-28-2016, 03:48 AM   #10
Michael Uplawski
Member
 
Registered: Dec 2015
Location: Normandy, France
Distribution: Debian buster/sid
Posts: 539
Blog Entries: 18

Original Poster
Rep: Reputation: 353Reputation: 353Reputation: 353Reputation: 353
Good morning.

I have still not succeeded in explaining the renaming mechanism. It is necessary only for piped-in content, which you must imagine like a data-stream and not a file being opened. So this data, which originates in email-attachments is not a file and does not come from a file. It is just data that I have to name, because the ruby module does not content with the mime-type, but wants a file extension, no matter what. I consider the behavior of the Roo-gem in this respect a little dumb, and imagine, that it has been developed with web-applications in mind (Ruby on Rails, Sinatra and such stuff). Should I ever arrive at having a github account, I might launch a feature request for the Roo gem.

Quote:
Originally Posted by grail View Post
When you come across a file with the incorrect file-extension, you then go through the process of renaming this to make the process work.
Which, if I understand correctly, will then involve creating a temp file on your system.
While this is automatically done in the current version of my program, ill-named files will not arrive often enough to justify on their own such a procedure.

Quote:
On the small reading I have done on mutt you could implement the same idea for your macro by copying the attachment
to /tmp and then performing any necessary tasks, including calling you ruby program on the file. Just a thought as this would solve the pipe dilemma currently vexing you
This on the other hand, is a possibility that I have to evaluate. The alternative is just deactivation of the interactive menu, in case that content comes from a pipe rather than an ordinary file. If, in a Mutt-macro I am able to pipe an attachment into a file and subsequently execute a second external tool in the same macro, I should either give this a try or write a shell-script to rid myself of the trouble...

Another good idea, Grail. Thank you.

Michael

Last edited by Michael Uplawski; 08-28-2016 at 06:44 AM. Reason: Kraut2English
 
Old 08-28-2016, 04:58 PM   #11
Michael Uplawski
Member
 
Registered: Dec 2015
Location: Normandy, France
Distribution: Debian buster/sid
Posts: 539
Blog Entries: 18

Original Poster
Rep: Reputation: 353Reputation: 353Reputation: 353Reputation: 353
Here is the final Macro definition for the Mutt Mail-User-Agent (or Transfer-Agent according to the configuration), which opens spreadsheet attachments with “viewworkbook.rb”:

Code:
macro attach A "<save-entry><kill-line>/tmp/mutt_at<enter><shell-escape>!viewworkbook.rb /tmp/mutt_at<enter>" "open attached spreadsheet in viewworkbook"
I could not find some of the details in the documentation and arrive at a working macro only by trial and error:
1.) You can combine mutt-internal commands with shell-commands in one and the same macro-definition. I ventured that this is possible, but the fact as such is not mentioned anywhere. Either a different cultural background or “technology-blindness” on the side of the authors of the documentation are responsible for my ignorance.
2.) Even with the <shell-escape> tag, which appears to be necessary, you must use bang-syntax for the actual call to “viewworkbook”. At this point, I am just happy with the result and do not ask any more questions... be it that way!

Conclusion: There is no more need to pipe-in content to the viewworkbook program. As also the mailcap entry for some spreadsheet formats has no effect, I consider a macro, like the one above, the best way to hand over mail-attachments to external applications.

The viewworkbook Ruby-gem will be updated one of these days and cleaned of the, now obsolete, pipe-in mechanism.

Last edited by Michael Uplawski; 08-28-2016 at 04:59 PM.
 
Old 08-28-2016, 07:39 PM   #12
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,552

Rep: Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899Reputation: 2899
I found similar to your macro on my searching, but forgot to copy it in <oops>

http://stackoverflow.com/questions/2...t-email-client
 
Old 08-29-2016, 11:55 AM   #13
Michael Uplawski
Member
 
Registered: Dec 2015
Location: Normandy, France
Distribution: Debian buster/sid
Posts: 539
Blog Entries: 18

Original Poster
Rep: Reputation: 353Reputation: 353Reputation: 353Reputation: 353
Quote:
Originally Posted by grail View Post
I found similar to your macro on my searching, but forgot to copy it in <oops>

http://stackoverflow.com/questions/2...t-email-client
Don't worry, I had found the same article but could not make my own version work, probably because of the the missing '!'
By the way, I found out, that with the newer Spreadsheet-formats xlsx even the altered SoftMaker pmdx file-type is now readable with Roo... The latter does not even conform to the Microsoft® OOXML standard. I should feel uneasy, as this is not really comprehensible... but it works.

I will write another Blog-entry, but do not yet know under which title... viewworkbook, Mutt-Macros.., “File-Magic and the magic-file“ or whatever. Guess I'll just type away and set the title afterwards.

Edit: New Blog Text-mode spreadsheet viewer

Edit II: I was quick to state incompatibilities with the Microsoft® OOXML-standard, but I have learned in the meantime that, in combination with the word “Microsoft”, the word standard does not make a lot of sense for OOXML... I put it in the same bin with HTML5.
Attached Thumbnails
Click image for larger version

Name:	maybe_html5.png
Views:	1
Size:	2.5 KB
ID:	22951  

Last edited by Michael Uplawski; 09-06-2016 at 03:29 AM. Reason: Blog
 
  


Reply

Tags
ioctl, raw, ruby, spreadsheet, stdin


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Read from stdin in man? dansimon Linux - General 6 06-18-2013 03:56 AM
Piping stdin/out/err from a subprocess default5 Programming 11 03-31-2009 08:59 PM
piping variables from stdin ziox Programming 6 02-13-2005 07:56 PM
Telnet user redirectng/piping stdin/out? MikHud Linux - General 2 02-11-2002 05:18 AM
Telnet user redirectng/piping stdin/out? MikHud Linux - Networking 0 01-30-2002 11:55 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 03:05 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration