cURL for php

Mr. Alex · 09-16-2011, 06:57 AM

Hello! Please explain me why do I (or someone else) need to use curl in php? They say I can download web-page with it. But we have file_get_contents() for this. Why use curl? I can't understand even after reading info in Web. Just don't get it.

PatrickDickey · 09-16-2011, 07:22 AM

Quote:

Originally Posted by Mr. Alex

Hello! Please explain me why do I (or someone else) need to use curl in php? They say I can download web-page with it. But we have file_get_contents() for this. Why use curl? I can't understand even after reading info in Web. Just don't get it.

The little that I read about this indicates that cURL is faster at downloading the pages. That's about the only real benefit that I've found. This might help you a bit (at least the code example might)

Please note that I'm not a PHP programmer. So, someone with more experience in this language will most likely correct (or clarify) my answer.

Have a great day

Patrick.

Mr. Alex · 09-16-2011, 07:57 AM

Thanks PatrickDickey!
Someone in the comments said:
"using curl we can post, put and also get. but file_get_contents is only get."

I was searching YouTube to find something interesting about curl and found tutorial about getting info: http://youtu.be/PvEJz6du7R0 . That looks like you must write very different code for each site (maybe depending on it's structure), Alex Garrett doesn't seem to understand curl well. So can someone who codes PHP create a curl code in PHP to show an example how to log into this LQ.org forum and post some message?

Snark1994 · 09-16-2011, 08:20 AM

Also, a lot of (shared) servers that I've used disable the file_get_contents() function, but you can then use curl to request web pages.

Here's an example from an e-commerce website I built, it checks for stock availability using an enquiry URL:

Code:

function stockAvailable($productModel,$stockType){
      $productModel = substr($productModel,3);
      if($stockType === "MADISON"){
          $ch = curl_init();
          curl_setopt($ch,CURLOPT_URL, "http://www.madisonb2b.co.uk/stockenquiry.aspx?code=".$productModel);
          curl_setopt($ch,CURLOPT_FRESH_CONNECT,TRUE);
          curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,5);
          curl_setopt($ch,CURLOPT_RETURNTRANSFER,TRUE);
          curl_setopt($ch,CURLOPT_TIMEOUT,10);
          $html=curl_exec($ch);
          if($html==false){
              $m=curl_error(($ch));
              error_log($m);
              echo($m);
              return false;
          }
          curl_close($ch);
          $madisonInStock = htmlspecialchars($html);
          return (substr($madisonInStock,0,3) === "YES");
      } else if($stockType === "DRAPER") {
          if(!file_exists("/tmp/draperStock.csv") || (filemtime("/tmp/draperStock.csv") - time() > 24*60*60)){
              //if draperStock file doesn't exist or is more than a day out of date...
              $ch = curl_init();
              curl_setopt($ch,CURLOPT_URL,"http://www.drapertools.com/products/stock.csv");
              $out = fopen('/tmp/draperStock.csv', 'w');
              curl_setopt($ch, CURLOPT_FILE, $out);
              curl_exec ($ch);
              curl_close ($ch);
              fclose($out);
          }
          $in  = fopen('/tmp/draperStock.csv', 'r');
          while (!feof($in)){
              $buffer = fgets($in, 4096);
              if(substr($buffer,1,5) === $productModel){
                  if(substr($buffer,9,1) === "Y"){
                      return true;
                  }
              }
          }
          fclose($in);
          return false;
      } else if ($stockType === "NONE") {
          return true;
      } else {
          return false;
      }
  }

It uses curl two different ways, one writes it to a file, one just examines the contents of the return result.

EDIT: Very sorry, didn't read your post properly. I'll have a go at your posting problem when I get back from work and Muay Thai. I hope the code I posted helps in some way, though...

Hope this helps,

Proud · 09-16-2011, 09:13 AM

curl can do SSL and cookie handling, imagine having to log in and handle session authentication cookies to be able to actually retrieve the contents of a deeper page from a site, or submit a restricted (POST) request which could span several pages of interaction manually.

As for having to code specifically for each site, many will have sane features to prevent XSS and CSRF attacks so you'll need to be able to provide things like dynamic hidden token form elements as well the page/request-specific variables/parameters. But I believe that goes for file_get_contents and anything else that's automating/scraping site interaction, not a curl-specific issue.

Snark1994 · 09-16-2011, 04:26 PM

Had a brief go at it; I was prototyping in Python, if you want the code then I will post it. However, the real challenge is not how to use curl, it's the sending of the correct headers and contents (as Proud mentioned), so my writing code to log into LQ will not really help you with your problem, unless your problem is to spam LQ :P

If you are trying to log in automatically to a site, you're going to need to have some software to look at the headers/content that your browser is sending - I've used fiddler before (on windows) and the 'Net' panel in firebug (on Linux)

Hope this helps,