LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 01-27-2012, 04:26 PM   #16
Cyrolancer
Member
 
Registered: Jan 2012
Distribution: Debian
Posts: 52

Original Poster
Rep: Reputation: Disabled

I will surely read and consider the information you have given in this topic. I am going to play on this script for a few days and then incorporate into production use in my company.

Additionally, I am going to handle some basic error reporting using the output of this python script. I think this is nothing to do with the working principle of this script, I just need the outputs.

For example, I tried to retrieve http://localhost.fmd url. As you know, there is no such an url The script output is:

Quote:
Cannot resolve hostname (http://localhost.fmd)
And another example: I tried to load a MP3 from this script. The output is:

Quote:
Plugin will handle load (http://mp3site.com/download.mp3)
These are nothing to do with this script and I think these are the outputs of the WebKit. Let's think and be rational: This script doesn't need to handle such outputs. But as I am passing the output of this script to another script (such as a PHP or BASH script), the status code you have given in -vv parameter is very important and also readable by anyone. Also it would be good to what errors or warnings WebKit can show during or after the execution of this script, such as the ones listed above.

Do you know any reference to these error or warning codes? Then, I could implement them in my PHP or BASH script.
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 01-27-2012, 10:21 PM   #17
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 947Reputation: 947Reputation: 947Reputation: 947Reputation: 947Reputation: 947Reputation: 947Reputation: 947
Quote:
Originally Posted by Cyrolancer View Post
I tried to retrieve http://localhost.fmd url.
The image creator function returns a tuple containing width, height, format. I was utterly lazy, and just report errors by setting width and height to zero, and saving the error message in format. Here:
Code:
    def _failed(self, view, frame, uri, gerror):
        import gtk
        import ctypes
        msg = ctypes.cast(int(str(gerror)[13:-1],16)+8, ctypes.POINTER(ctypes.c_char_p))[0]
        self.image = (0, 0, "%s (%s)." % (msg, uri))
        gtk.main_quit()
This gets called if the page load fails. It does not notice if you get an error page from the remote server, only if the URL won't work. The msg string is the one webkit returns. (I shamelessly stole it from somewhere, it's pretty cryptic!) In the self.image line, I set the actual string that will be output later on to "message (URL used).".

Quote:
Originally Posted by Cyrolancer View Post
These are nothing to do with this script and I think these are the outputs of the WebKit.
Well, the actual error message is from WebKit, but I did write the bit on how it'll output it. Also note that it will return a nonzero exit status if an error message is output. You can catch that using e.g.
Code:
if ! python .... ; then
     oops, did not get an image
fi
or checking the last exit status code, $?, immediately after running the python command.

Quote:
Originally Posted by Cyrolancer View Post
Do you know any reference to these error or warning codes?
Well, the error messages are pretty eclectic, string content-wise, but the Python code could try to determine at least the cases you think are important to differentiate.

Then, the Python script could return a nonzero exit status that tells the error cause. Exit statuses 1 to 127 are available for just this purpose. They're not really standardized, other than that they're all errors. (Almost all scripts and utilities just return 1 if an error occurs, and of course 0 when successful -- zero is the success exit status code.)

In Bash, the exit status is available as a number in $? immediately after running the command.

In PHP, the exit status is saved in the optional third parameter to the exec() function, and returned by the pclose() and proc_close() functions (whichever of the three approaches you use to call the Python code).

If you care to play a bit with the script to see which of the error cases you want/need/must differentiate, I think I could rewrite the script to make it cleaner and easier to understand. I can reformat the output so it'll be easier to use in a PHP script -- I myself would like to then get the result as a nice associative array. I haven't used WebKit before, and you really can tell by looking at the script; I think I'd sleep better, if I didn't leave it in such a sorry state...
 
Old 01-27-2012, 10:51 PM   #18
Cyrolancer
Member
 
Registered: Jan 2012
Distribution: Debian
Posts: 52

Original Poster
Rep: Reputation: Disabled
I have never used Python before like in this project. It seems that my questions are going out of the scope of this script and becoming silly than ever. I need to work on programming with Python and get some experience with it, and then I can possibly ask questions or modify this script. So, it is better to stop at this point. No need further working on this script, it simply converts URL to PNG and that's all I want, at least for now The additional modifications should be my homework from now on If I manage to do something useful, I will gladly share it in this forum.

Thank you for all your support You are awesome!

Last edited by Cyrolancer; 01-27-2012 at 11:01 PM.
 
Old 01-27-2012, 11:32 PM   #19
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 947Reputation: 947Reputation: 947Reputation: 947Reputation: 947Reputation: 947Reputation: 947Reputation: 947
If I ever find I need to snapshot more than an occasional web page, I'll convert this script into a daemon.

Most of the overhead in the script is from the fact that it starts up Xvfb and a WebKit browser instance for each invocation. That is unnecessary, when a lot of images are converted. Instead, the browser instance could be kept, and it and Xvfb only torn down when there has been inactivity for a while. The script would become a simple daemon, saving the snapshots (and perhaps thumbnails of the snapshots) into a dedicated output directory.

It is possible to set up the directory so that each client can read their own file, but cannot list the contents of the directory. (Having world-readable files, in a non-readable directory, with long random file names, makes it pretty much impossible to guess the names of the other files.) It would be easy to set the directory up so that the clients can copy any images elsewhere if they want to keep them, but otherwise the daemon will remove them in say fifteen minutes.

The daemon would use a Unix domain datagram socket to receive requests and respond with the image name, details, and conversion status. Frontends for this would be trivial to write, even using Bash only. It would suit perfectly for using via a PHP script on a web page.

I haven't yet written such in Python (but a number of similar daemons in C), and there are certain security issues I'd like to emphasize; they seem to be overlooked too often, leading to insecure services. I think it might be a good exercise and example, too. If you find it interesting enough, perhaps we could develop it together here, in a new thread. I'll be happy to provide code, if someone (*cough* you) does the diligent testing and provides feedback.

Last edited by Nominal Animal; 01-27-2012 at 11:33 PM.
 
Old 01-28-2012, 04:17 AM   #20
Cyrolancer
Member
 
Registered: Jan 2012
Distribution: Debian
Posts: 52

Original Poster
Rep: Reputation: Disabled
"deamon" idea is great Especially, I have found that my previous script (using poppler and imagemagick) does not suit "all" of the webpages I am trying manage. It works great on some pages but it still needs to be worked on to fit world wide web. (Well, I am still experimenting on that previous script ) Also, it produces a considerable amount of load on the server, when processing 100+ web pages.

In a few weeks, I am going to build a testing web front-end to this python script. Also, I have started reading Python books and experimenting on much simpler codes. If I manage to learn a bit of Python, I am willing to help you by writing code. Moreover, I will be glad to test the code you have provided in my experimental web front-end.

Last edited by Cyrolancer; 01-28-2012 at 05:03 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
64 bits or 32 bits on AMD athlon x2 ram 4gb? albertoburgos Linux - Newbie 13 08-15-2011 03:37 PM
What is the meaning of 32 bits x 16 bits in a flash memory? archieval Linux - Embedded & Single-board computer 4 05-25-2011 02:37 PM
Things to consider when compiling C using 32 bits instead of 64 bits processors rpomerleau Programming 2 07-08-2008 01:22 PM
How to change MAC address from 48 bits to 32 bits? cywong.digi Linux - Networking 1 09-05-2007 09:28 AM
32 bits version distros running 64 bits CPU javb Linux - General 4 04-02-2006 07:21 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 10:51 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration