LinuxQuestions.org - [SOLVED] [Ruby] Cannot read from STDIN after piping-in a file

- Programming (https://www.linuxquestions.org/questions/programming-9/)

- - [Ruby] Cannot read from STDIN after piping-in a file (https://www.linuxquestions.org/questions/programming-9/%5Bruby%5D-cannot-read-from-stdin-after-piping-in-a-file-4175588098/)

[Ruby] Cannot read from STDIN after piping-in a file

Good morning.

I need a hint on reading from STDIN and the IOCTL options. This may or may not be Ruby-specific. Sorry for the long text. The whole code is available, selected code-sections are quoted below.

Context: I wrote a small Ruby-program to visualize multi-spreadsheet
workbooks in a terminal. Deriving the mime-type of the spreadsheet content
allows me to choose a wrapper-instance adapted for ODS, XLS or XLSX-files. The
external module (“Ruby-Gem” Roo) which does most of the work, however
insists on the file-extension to correspond to the type of the chosen
wrapper.., meaning: I know from the mime-type, that I need an ODS-wrapper,
but when my file does not have an *.ods extension, an exception is raised and
that's it.

This is normally not a problem. Calls like the following

Code:

user@machine:~/[path]/viewworkbook_trunk/test$ ../bin/viewworkbook.rb ./some_spreadsheet_file.xls

produce output like in the screen shot, below.

Edit: The issue with the file-extension is just an inconvenience
and *not* the core problem that I seek help for in this discussion. It
does, though, impose code which interferes with the function that pipes-in
data from STDIN.

In the screen shot, PSE note the line of options,
below the table. The program is interactive, in that it lets the user move
through the sheets in the workbook, by name or number, adapt the width of the
table-columns or save a single table or the whole workbook to a new file.

Reproducing the error: This program shall be used in a macro to quickly open
email-attachments from Mutt, my email-client. For the macro, I must be able to
pipe-in the file-content to the Ruby-program. As the wrapper class only works
with files of the right filename-extension, I must first redirect the
spreadsheet content to a temporary file, give it the right extension, then
open it just like any other spreadsheet-file.

Here my program crashes after the generation of the table. in the terminal, I
see the complete, first spreadsheet from the workbook and the user options
(like in the screen shot). The following exception is raised when the program
starts to wait for user-input:

Code:

/[path]/viewworkbook_trunk/lib/user_input.rb:30:in `raw': Inappropriate ioctl for device (Errno::ENOTTY)

        from /[path]/viewworkbook_trunk/lib/user_input.rb:30:in `wait_for_user'

        from /[path]/viewworkbook_trunk/lib/menu.rb:80:in `show'

        [...]

I show you some of what I consider relevant code because it reads from STDIN.
As the difference in the working and the failing calls to my program is in the
way, that spreadsheet content is read, there is clearly something missing in
my handling the standard input device, and probably something which is
important in any such routine, independent of the programming language used.

Code
Here is the code from the main class, which “pipes in” spreadsheet content:

Code:

        # verify that the programm argument is '-' 

        if args.length == 1 && args[0] == '-'

                        # not all the code is really needed for 

                        # the current function to work. But I leave it here

                        # for authenicity.

                        args.compact!

                        args.shift

                        # First, I verified, that STDIN can be read.

                        # @log.debug('file piped-in: ' << $<.read)

                        tfile = nil

                        # Verify that there is data on STDIN

                        if(!$<.eof?)        

                                @log.debug('reading from STDIN')

                                # ... provide a temporary file in that case,

                                tfile = Tempfile.new('spreadsheet_temp')

                                File.open(tfile, 'w+') {|of| of << $<.read}

                                @log.debug("Temporary file %s is written." %File.path(tfile))

                                

                                wb = tfile

                        end

                end

After this, the workbook is entirely stored in the file ‘wb’ which still has
no file-extension, but is named something like
“/tmp/spreadsheet_temp20160827-5802-1s6jlg1”.

For the recognition of spreadsheet types, I defined the Mime-types for use
with the magic library, via the Ruby Gem “FileMagic”.

Code:

ODS_MIME = "application/vnd.oasis.opendocument.spreadsheet"

XLS_MIME = "application/vnd.ms-excel"

XLSX_MIME = "application/zip"

CSV_MIME = "text/plain"

The wrapper is generated and the file-extension is added to the temporary
file, if needed in a (“static”) factory class.

Code:

        def self::workbook(file)

                @@file = file

                if !@@workbook

                        begin

                                @@workbook = case mime_type

                                            when ODS_MIME 

                                                    @@log.debug('ODS')

                                                    unless File.extname(@@file) == '.ods'

                                                            odsfile = @@file.dup << '.ods'

                                                            File.rename(@@file, odsfile) 

                                                            @@file = odsfile

                                                    end

                                                    # create the OpenOffice instance

                                                    Roo::OpenOffice.new(File.path(@@file)

                                              when XLS_MIME

                                                (...)

                                              when XLSX_MIME

                                                (...)

Then, the menu is created below the table, and we wait for user input, when
the program crashes:

Code:

require 'io/wait'

require 'io/console'



def wait_for_user()

        @log.debug('wait_for_user() ')

        char = STDIN.raw(&:getc)        



# Here is another way to code the non-blocking 

# read from STDIN which usually works, but fails in

# the exact same way after a spreadsheet has been

# piped-in 



=begin

        char = nil

        STDIN.raw do 

                STDIN.noecho do

                        until (STDIN.ready?)

                                sleep(0.1)

                        end



                        char = (STDIN.read_nonblock(1).ord rescue nil)

                end

        end

=end

        return char

end

The whole program in its current development-version 0.0.5 is available as a Ruby-gem,
containing all the code, from rubygem.org, either from the site:
https://rubygems.org/gems/viewworkbook
or by installing it with the gem-tool:

Code:

gem install viewworkbook

This is enough to see the complete code. To run the program, you also need the
Roo and the FileMagic gems.

I am still having a bit of a look at things, but may I ask what version of ruby are you using? I have seen a number of these types of errors related to 1.9.3, so it may be nothing but thought I should ask before going too much further :)

Quote:

Originally Posted by Michael Uplawski (Post 5596765)

This is normally not a problem. Calls like the following

Code:

user@machine:~/[path]/viewworkbook_trunk/test$ ../bin/viewworkbook.rb ./some_spreadsheet_file.xls

So, is the failing case equivalent to something like this?

Code:

user@machine:~/[path]/viewworkbook_trunk/test$ ../bin/viewworkbook.rb < ./some_spreadsheet_file.xls

In which case the problem is essentially that you can't read user input from stdin if stdin is already being used as a file pipe.

Quote:

Originally Posted by grail (Post 5596838)

Thank you for the idea. Even if I use ruby 2.4.0dev (from the trunk, build on 21/8), I shall install a somewhat “stable” version just to exclude dumb coincidences... ;-) (What's “unstable” anyway, and should I put a trema on coïncidences?)

(Joking)

Michael

Quote:

Originally Posted by ntubski (Post 5596859)

So, is the failing case equivalent to something like this?

Code:

user@machine:~/[path]/viewworkbook_trunk/test$ ../bin/viewworkbook.rb < ./some_spreadsheet_file.xls

In which case the problem is essentially that you can't read user input from stdin if stdin is already being used as a file pipe.

In deed, I deem my cat spreadsheet.(xls|xlsx|ods) | viewworkbook.rb - equivalent to your alternative command line.
However, are you telling me, that there cannot be a program which continues listening on STDIN after ... WAIT!! I see: Do I understand correctly, that the pipe is opened before the execution of my program and there is just nothing that I can do do close it “during” the execution of the program? ... That would be an occasion to pay a visit to the Calva distillery in the neighborhood.

I think your summation is correct but will let ntubski confirm.

I had 2 questions though:

1. Why would you prefer the piped / redirected version over simply calling your program with the file?

2. If you took out the redirection process wouldn't it be simpler to test the file and rename accordingly (temp file if need be) on the actual file and the open it inside your ruby code? (more curious than anything)

Quote:

Originally Posted by grail (Post 5596963)

1. Why would you prefer the piped / redirected version over simply calling your program with the file?

This redirection would be needed to pipe in attachments from my mail-client (Mutt). A macro command can be defined, like one of those, that I use already:

Code:

macro index M "|maillinks -\n"          "create link-list"

macro pager M "|maillinks -\n"          "create link-list"

This pipes a html-mail to another program “maillinks” which shows me the URLs of included html-links in a numbered list. I imagine a similar macro to view spreadsheets that I receive as mail-attachment:

Code:

macro attach <F12> "|viewworkbook.rb -\n" "show spreadsheet"

Without the pipe, I would have to save the attachment first to a file and it would make the call to viewworkbook superfluous as I can run a full-blown spreadsheet program on the file right away...

Quote:

2. If you took out the redirection process wouldn't it be simpler to test the file and rename accordingly (temp file if need be) on the actual file and the open it inside your ruby code? (more curious than anything)

When I do not use the pipe but call the program with a spreadsheet as argument, no renaming is necessary, as long as the files have the correct file-extension.

Thanks for asking. I reconsider all my decisions, each time that a question is formulated in a different way than mine. ;)

Just for completeness, there is a second use for this programming exercise. Versions of the SoftMaker Office Suite before 2015 stored spreadsheets in a format PMD which identifies as “Composite Document File V2 Document, Little Endian”, i.e. the same as XLS. As this office suite does not come with a scripting interface, I hope to manipulate these files anyway from outside the spreadsheet program. There is no connection between this aspect and the pipe-trouble. Even from inside the SoftMaker program, I cannot call external programs on a current file but must open it in a separate process.

Quote:

Originally Posted by Michael Uplawski (Post 5596958)

In deed, I deem my cat spreadsheet.(xls|xlsx|ods) | viewworkbook.rb - equivalent to your alternative command line.

Yup.

Quote:

However, are you telling me, that there cannot be a program which continues listening on STDIN after ... WAIT!! I see: Do I understand correctly, that the pipe is opened before the execution of my program and there is just nothing that I can do do close it “during” the execution of the program?

You can close it, but then you just have a closed pipe. Nothing you can do with that. The real problem is that you never get the file handle of the terminal that would let you talk to the user. Maybe you can try opening "/dev/tty"?

Thanks for all the information ... I wasn't aware just how much mutt could do :)

Your information did give me one side thought though. When you come across a file with the incorrect file-extension, you then go through the process of renaming this to make the process work.
Which, if I understand correctly, will then involve creating a temp file on your system. On the small reading I have done on mutt you could implement the same idea for your macro by copying the attachment
to /tmp and then performing any necessary tasks, including calling you ruby program on the file. Just a thought as this would solve the pipe dilemma currently vexing you :)

Good morning.

I have still not succeeded in explaining the renaming mechanism. It is necessary only for piped-in content, which you must imagine like a data-stream and not a file being opened. So this data, which originates in email-attachments is not a file and does not come from a file. It is just data that I have to name, because the ruby module does not content with the mime-type, but wants a file extension, no matter what. I consider the behavior of the Roo-gem in this respect a little dumb, and imagine, that it has been developed with web-applications in mind (Ruby on Rails, Sinatra and such stuff). Should I ever arrive at having a github account, I might launch a feature request for the Roo gem.

Quote:

Originally Posted by grail (Post 5597013)

When you come across a file with the incorrect file-extension, you then go through the process of renaming this to make the process work.
Which, if I understand correctly, will then involve creating a temp file on your system.

While this is automatically done in the current version of my program, ill-named files will not arrive often enough to justify on their own such a procedure.

Quote:

On the small reading I have done on mutt you could implement the same idea for your macro by copying the attachment
to /tmp and then performing any necessary tasks, including calling you ruby program on the file. Just a thought as this would solve the pipe dilemma currently vexing you :)

This on the other hand, is a possibility that I have to evaluate. The alternative is just deactivation of the interactive menu, in case that content comes from a pipe rather than an ordinary file. If, in a Mutt-macro I am able to pipe an attachment into a file and subsequently execute a second external tool in the same macro, I should either give this a try or write a shell-script to rid myself of the trouble...

Another good idea, Grail. Thank you.

Michael

Here is the final Macro definition for the Mutt Mail-User-Agent (or Transfer-Agent according to the configuration), which opens spreadsheet attachments with “viewworkbook.rb”:

Code:

macro attach A "<save-entry><kill-line>/tmp/mutt_at<enter><shell-escape>!viewworkbook.rb /tmp/mutt_at<enter>" "open attached spreadsheet in viewworkbook"

I could not find some of the details in the documentation and arrive at a working macro only by trial and error:
1.) You can combine mutt-internal commands with shell-commands in one and the same macro-definition. I ventured that this is possible, but the fact as such is not mentioned anywhere. Either a different cultural background or “technology-blindness” on the side of the authors of the documentation are responsible for my ignorance.
2.) Even with the <shell-escape> tag, which appears to be necessary, you must use bang-syntax for the actual call to “viewworkbook”. At this point, I am just happy with the result and do not ask any more questions... be it that way!

Conclusion: There is no more need to pipe-in content to the viewworkbook program. As also the mailcap entry for some spreadsheet formats has no effect, I consider a macro, like the one above, the best way to hand over mail-attachments to external applications.

The viewworkbook Ruby-gem will be updated one of these days and cleaned of the, now obsolete, pipe-in mechanism.

I found similar to your macro on my searching, but forgot to copy it in <oops>

http://stackoverflow.com/questions/2...t-email-client

Quote:

Originally Posted by grail (Post 5597540)

I found similar to your macro on my searching, but forgot to copy it in <oops>

http://stackoverflow.com/questions/2...t-email-client

Don't worry, I had found the same article but could not make my own version work, probably because of the the missing '!'
By the way, I found out, that with the newer Spreadsheet-formats xlsx even the altered SoftMaker pmdx file-type is now readable with Roo... The latter does not even conform to the Microsoft® OOXML standard. I should feel uneasy, as this is not really comprehensible... but it works.

I will write another Blog-entry, but do not yet know under which title... viewworkbook, Mutt-Macros.., “File-Magic and the magic-file“ or whatever. Guess I'll just type away and set the title afterwards.

Edit: New Blog Text-mode spreadsheet viewer

Edit II: I was quick to state incompatibilities with the Microsoft® OOXML-standard, but I have learned in the meantime that, in combination with the word “Microsoft”, the word standard does not make a lot of sense for OOXML... I put it in the same bin with HTML5.