I am using Libcurl to fetch http code for pages (only the status code in header, no body), but the performance is not so satisfying : some of the links would take around 1 seconds especially those from abroad. Of course, a best performance it has is 14 links for just 2 seconds (in google).

I am wondering if there is some other library could do better than Libcurl in this case ? Or it is reasonable for all to take 1 seconds just for http code for those remote links ?


I really need some more info to prove my boss whether there is better solution or not.

Any idea is well appreciated,



Thanks,

-Kun

Well, if you consider how fetching a page works then you will understand that there are not many tricks (except parallelism) which will do it faster. When your browser or your libcurl scripts try to fetch a page they:

1. Request from your DNS server, the IP corresponding to the name of the site you requested
2. Use the server's reply to open a socket to that IP, port 80
3. Send a small HTTP message describing what you want
4. Receive the html code

It is not a cheap process and I think the performance you are getting is fine. If you really want thing to be quicker then you will have to use more than one computers which will download parts of a URL list..

Edit: Well, I just realized that you don't necessarily need another computer. Just more threads

To speed things up a bit, you can cache the DNS responses.

Thanks a lot, I think I got it right even it is not satisfying.

I did try multi-threaded, and it was less than 5% faster for pages containing many links (>150) and even worse for fewer ones (<20).

Looks like it is the best Libcurl could do.

Have you tried libcurl's multi interface that I mentioned in your other thread? It allows concurrent transfers, probably with less overhead than threads (running a lot of threads at once can slow things down considerably).

I think libcurl already does this: http://curl.haxx.se/libcurl/c/libcur...ml#Persistence

Thanks, I gave up on the usage of multi-interface since it seems will change my whole programming structure.

Could you give me some number of estimation on a multi-interface performance ? For example, 200 threaded VS Multi-interface (http code request only)? I will try that if it really have a big difference.

You changed from single threaded to multi threaded without changing your program structure?

Sorry, haven't really written that many downloading programs. 200 threads does sound like a lot though, are you getting more than 90% cpu usage? It occurs to me that using threads with libcurl might prevent a lot of the optimizations mentioned in http://curl.haxx.se/libcurl/c/libcur...ml#Persistence, since each thread needs a separate handle.