[SOLVED] Python urllib finding the real URI of the start page

ocpaul20 · 05-02-2015, 10:09 PM

[SOLVED] - there is no way to determine this if the remote server does not send the index file name (which I was looking for) back.
===========================

I have a URL and I want to find out the file name of the initial page it is loading. Eg: index.php, index.htm, index.html, default.htm, or any of the other possible start pages available.

Here is my code but it only gives the same URL as I started with. Is there any way to find out the default starting page name for this site please? ( By trying various options I have found out it is index.php
So I want it to return http://synapse.ararat.cz/files/contrib/index.php )

Code:

import sys, os, urllib2

starturl = "http://synapse.ararat.cz/files/contrib/"
req = urllib2.Request(starturl) # , datagen, headers)
res = urllib2.urlopen(req)
finalurl = res.geturl()
print finalurl # http://synapse.ararat.cz/files/contrib/

Thanks for any help.

SoftSprocket · 05-03-2015, 10:48 AM

You could check the headers (res.info()) for a Location header but unless there is a redirect there probably won't be one.

dugan · 05-03-2015, 11:33 AM

Well, tell me how you found that it's index.php, and I'll tell you how to do that in Python.

As far as I can tell, though, that information just isn't programmatically available for that URL in particular. It's not in any part of the HTTP response headers, which I've checked using both Chromium Developer Tools and with urllib itself.

Web servers do URL-rewriting server-side. They take "files/contrib/" and convert them internally to "index/contrib.php", then they just send the response back. All the client (Chromium, urllib2, etc) sees is that it's making a request to "index/contrib" and getting a response back.

ocpaul20 · 05-03-2015, 09:54 PM

I found the index.php by trial and error - trying index.html, index.htm, index.php, default.html etc

If this information is not available in the headers or from the http server at the remote end, then maybe I have to determine if there is no file on the end of the url and then try a few likely candidates programatically. I know that many content management systems do not have a file name on the end due to them using page IDs instead which look bad as URLs. Anyway...

Thanks for the help and info guys.