[libcurl] disable output in terminal

Kunsheng · 10-23-2009, 09:05 PM

I am doing some libcurl program in windows xp with VC 2005 (Win 32 program):

program reads a list of urls from input file and try to check http code for each of them.

The program runs pretty slow (10-15 seconds to fetch code per url !) ,and libcurl keep output everything into terminal when I run the program, what's more, it seems the program is NOT just fetching HTTP code in header but the whole page content.

The details code is as below:

Code:

   CURL *curl_handle;
	  //CURLcode res;

  	  curl_global_init(CURL_GLOBAL_ALL);

	  curl_handle = curl_easy_init();
	  curl_easy_setopt(curl_handle,CURLOPT_URL,url);

     
      curl_easy_setopt(curl_handle, CURLOPT_USERAGENT, "libcurl-agent/1.0");
      curl_easy_setopt(curl_handle, CURLOPT_NOPROGRESS);

	  int curl_code = curl_easy_perform(curl_handle);

	  http_code = 0;

	  curl_easy_getinfo(curl_handle, CURLINFO_HTTP_CODE, &http_code);
	 
	  curl_easy_cleanup(curl_handle);

	  curl_global_cleanup();

I been running this code inside BOINC, which disables libcurl terminal output and 10 times faster than this libcurl program alone....

Any idea ?

Thanks,

-Kun

Also could some one delete these two post, I posted it in software by mistake:

http://www.linuxquestions.org/questi...ntents-764069/

http://www.linuxquestions.org/questi...rminal-764041/

Thanks,

ntubski · 10-24-2009, 11:44 AM

If you do

Code:

curl_easy_setopt(curl_handle, CURLOPT_NOBODY, 1);

libcurl will send a HEAD request so the page contents won't be served. You might gain even more speed if you use the multi interface which will let the program wait for all the urls at the same time.

Quote:

Originally Posted by Kenshung

The program runs pretty slow (10-15 seconds to fetch code per url !)

Hmm, I ran this on Google's home page and it took 0.148s with terminal output, page is pretty short though.

Quote:

Also could some one delete these two post, I posted it in software by mistake:

Use the report button on one of those posts (next to the quote button).

Kunsheng · 10-24-2009, 11:47 PM

It is pretty quick but seems not work when coming to 'amazon.com' .

I got code '405' when putting CURLOPT_NOBODY in the program but actually it is OK (should be 200).

Is it possible I place it in a wrong place ?

My code is as below:

Code:

int http_code = -1;
if(http_code < 0)
	{
	
       CURL *curl_handle;
	  //CURLcode res;

  	  curl_global_init(CURL_GLOBAL_ALL);

	  curl_handle = curl_easy_init();
	  curl_easy_setopt(curl_handle,CURLOPT_URL,url);

     
      curl_easy_setopt(curl_handle, CURLOPT_USERAGENT, "libcurl-agent/1.0");
      
	  curl_easy_setopt(curl_handle, CURLOPT_NOBODY, 1);  # is the position correct ?

	  int curl_code = curl_easy_perform(curl_handle);

	  http_code = 0;

	  curl_easy_getinfo(curl_handle, CURLINFO_HTTP_CODE, &http_code);
	 
	  curl_easy_cleanup(curl_handle);

	  curl_global_cleanup();

ntubski · 10-25-2009, 09:24 AM

Looking at the headers

Code:

 curl_easy_setopt(curl_handle, CURLOPT_HEADER, 1);


HTTP/1.1 405 MethodNotAllowed 
Date: Sun, 25 Oct 2009 14:04:16 GMT
Server: Server
Set-Cookie: skin=noskin; path=/; domain=.amazon.com; expires=Sun, 25-Oct-2009 14:04:16 GMT
x-amz-id-1: 1632JFKY77DHQZRC5CR0
allow: POST, GET 
x-amz-id-2: G5819ZOStmRQaUwqwGExnyGBWPC/P4Ut
Vary: Accept-Encoding,User-Agent
Content-Type: text/html; charset=ISO-8859-1

Amazon won't respond to HEAD requests. I suppose you'll have to replace the WRITEFUNCTION to ignore output:

Code:

size_t ignore_output( void *ptr, size_t size, size_t nmemb, void *stream)
{
    (void) ptr;
    (void) stream;
    return size * nmemb;
}

...
    curl_handle = curl_easy_init();

    curl_easy_setopt(curl_handle, CURLOPT_WRITEFUNCTION, &ignore_output);
...

Kunsheng · 10-26-2009, 08:06 AM

Looks like a stone could not kill 2 birds this time...

I may pick the 'header request only' for the speed and universal enable sites.

Thanks a lot!

Quote:

Originally Posted by ntubski

Looking at the headers

Code:

 curl_easy_setopt(curl_handle, CURLOPT_HEADER, 1);


HTTP/1.1 405 MethodNotAllowed 
Date: Sun, 25 Oct 2009 14:04:16 GMT
Server: Server
Set-Cookie: skin=noskin; path=/; domain=.amazon.com; expires=Sun, 25-Oct-2009 14:04:16 GMT
x-amz-id-1: 1632JFKY77DHQZRC5CR0
allow: POST, GET 
x-amz-id-2: G5819ZOStmRQaUwqwGExnyGBWPC/P4Ut
Vary: Accept-Encoding,User-Agent
Content-Type: text/html; charset=ISO-8859-1

Amazon won't respond to HEAD requests. I suppose you'll have to replace the WRITEFUNCTION to ignore output:

Code:

size_t ignore_output( void *ptr, size_t size, size_t nmemb, void *stream)
{
    (void) ptr;
    (void) stream;
    return size * nmemb;
}

...
    curl_handle = curl_easy_init();

    curl_easy_setopt(curl_handle, CURLOPT_WRITEFUNCTION, &ignore_output);
...