LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 01-02-2012, 06:05 AM   #1
BeaverusIV
Member
 
Registered: Oct 2004
Location: New Zealand
Distribution: Arch Linux 2010.05
Posts: 129

Rep: Reputation: 18
curl not downloading full webpage


I am trying to run a simple program to start learning curl, but it doesn't get the whole page, merely ~20KB of it :/

Code:
#include <iostream>
#include <string>
#include <curl/curl.h>

static std::string buffer;

static int writer(char *data, size_t size, size_t nmemb, std::string *writerData) {
	if(writerData == NULL) return 0;

	writerData->append(data, size * nmemb);

	return size * nmemb;
}

int main(int argc, char **argv) {
	CURL *curl;
	CURLcode res;

	curl = curl_easy_init();
	if(curl) {
		curl_easy_setopt(curl, CURLOPT_URL, "http://www.neopets.com/games/pyramids/index.phtml");
		curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writer);
		curl_easy_setopt(curl, CURLOPT_WRITEDATA, &buffer);

		res = curl_easy_perform(curl);
		std::cout << buffer;

		curl_easy_cleanup(curl);
	}

	return 0;
}
Any reason why it is doing this?
 
Old 01-02-2012, 06:55 AM   #2
BeaverusIV
Member
 
Registered: Oct 2004
Location: New Zealand
Distribution: Arch Linux 2010.05
Posts: 129

Original Poster
Rep: Reputation: 18
After a bit more investigation I have found it cuts out at the same 'place' on the webpage with different pages having different amounts of data before it.

Is it possible they put in a character that mucks it up on purpose?
 
Old 01-02-2012, 07:04 AM   #3
BeaverusIV
Member
 
Registered: Oct 2004
Location: New Zealand
Distribution: Arch Linux 2010.05
Posts: 129

Original Poster
Rep: Reputation: 18
After using the cli curl and getting the same thing, then trying wget to retrieve it I realised it redirects, so after adding in curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, true); into my program it now gets the full 56KB.
 
  


Reply

Tags
c++, curl


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] WGET issues downloading recursively from webpage chrisportela Linux - Software 5 06-22-2011 06:07 PM
[SOLVED] curl -K command outputs weird symbols instead of downloading URL from file meridionaljet Linux - Newbie 7 05-16-2011 09:13 PM
wget will not download full webpage with images hedpe Linux - Software 2 02-16-2006 12:46 AM
Mozilla scroll bar inactive during downloading of a webpage Linils Linux - Software 0 03-19-2004 12:35 PM
how to invoke Mozilla webpage in full screen when it is run taoweijia Programming 5 02-21-2004 05:45 PM


All times are GMT -5. The time now is 07:38 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration