LinuxQuestions.org - [SOLVED] curl not downloading full webpage

- Programming (https://www.linuxquestions.org/questions/programming-9/)

- - curl not downloading full webpage (https://www.linuxquestions.org/questions/programming-9/curl-not-downloading-full-webpage-921586/)

curl not downloading full webpage

I am trying to run a simple program to start learning curl, but it doesn't get the whole page, merely ~20KB of it :/

Code:

#include <iostream>

#include <string>

#include <curl/curl.h>



static std::string buffer;



static int writer(char *data, size_t size, size_t nmemb, std::string *writerData) {

        if(writerData == NULL) return 0;



        writerData->append(data, size * nmemb);



        return size * nmemb;

}



int main(int argc, char **argv) {

        CURL *curl;

        CURLcode res;



        curl = curl_easy_init();

        if(curl) {

                curl_easy_setopt(curl, CURLOPT_URL, "http://www.neopets.com/games/pyramids/index.phtml");

                curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writer);

                curl_easy_setopt(curl, CURLOPT_WRITEDATA, &buffer);



                res = curl_easy_perform(curl);

                std::cout << buffer;



                curl_easy_cleanup(curl);

        }



        return 0;

}

Any reason why it is doing this?

After a bit more investigation I have found it cuts out at the same 'place' on the webpage with different pages having different amounts of data before it.

Is it possible they put in a character that mucks it up on purpose?

After using the cli curl and getting the same thing, then trying wget to retrieve it I realised it redirects, so after adding in curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, true); into my program it now gets the full 56KB.