Need help to use sock proxy while sending http request using AnyEvent in perl

unclesamcrazy · 05-11-2018, 10:10 AM

I am trying to send multiple http get requests using perl. I need to send those request using sock proxy.
If i use following code, I am able to send request with sock proxy

Code:

#!/usr/bin/perl
use strict;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new(
  agent => q{Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; YPC 3.2.0; .NET 
CLR 1.1.4322)},
);
$ua->proxy([qw/ http https /] => 'socks://localhost:9050'); # Tor proxy
#$ua->cookie_jar({});
$a = 10;
while( $a < 20 ) {
my $rsp = $ua->get('http://domain.com/type?parameter=1&parameter=2');
print $rsp->content;
$a = $a + 1;
}

It works successfully but I need to use AnyEvent to send multiple get requests parallely.

Code:

#!/usr/bin/perl
use strict;
use AnyEvent;
use AnyEvent::HTTP;
use Time::HiRes qw(time);
use LWP::Protocol::socks;
use AnyEvent::Socket;
my $cv = AnyEvent->condvar( cb => sub {
    warn "done";
});
my $urls = ["http://url-1-withallparameters",
"http://url-2-withallparameters",
"http://url-3-withallparameters",
];
my $start = time;
my $result;
$cv->begin(sub { shift->send($result) });
for my $url (@$urls) {
    $cv->begin;
    my $now = time;
    my $request;  
    $request = http_request(
      GET => $url, 
      timeout => 2, # seconds
      sub {
        my ($body, $hdr) = @_;
	if ($hdr->{Status} =~ /^2/) {
          push (@$result, join("\t", ($url,
				      " has length ",
				      $hdr->{'content-length'}, 
				      " and loaded in ",
				      time - $now,
				      "ms"))
	       );
        } else {
	  push (@$result, join("",
			       "Error for ",
			       $url,
			       ": (", 
			       $hdr->{Status}, 
			       ") ", 
			       $hdr->{Reason})
		);
	}
        undef $request;
        $cv->end;
      }
   );
  }
$cv->end;
warn "End of loop\n";
my $foo =   $cv->recv;
print join("\n", @$foo), "\n" if defined $foo;
print "Total elapsed time: ", time-$start, "ms\n";

This is working fine but I am not able to send these requests using sock proxy.
Even in terminal, I export proxy, commands like curl wget work fine with sock proxy but when I use perl command to execute this script as a sock proxy, it does not work. I could integrate it with LWP agent, but it is not working with Any Event.

I have used
proxy => [$host, $port],
below
GET => $url,
but it works for http and https proxy only, not for sock proxy. I am not sure what I am doing wrong. If you suggest any solution, it will be great help from your side.

Thanks
Sam

coralfang · 05-11-2018, 11:51 AM

I'm not sure what problem is caused with AnyEvent, but this may be useful, i like to use IO::Socket::Socks for any socks proxy connections;

You'd use it like this for HTTP requests:

Code:

#!/usr/bin/env perl
use strict;
use warnings;
use IO::Socket::Socks;
$|++;

# set the socks proxy
my ($socks_addr, $socks_port) = ("127.0.0.1", 9050);

my $sock = new IO::Socket::Socks(
	ProxyAddr	=> $socks_addr,
        ProxyPort	=> $socks_port,
        ConnectAddr	=> 'checkip.dyndns.org',
	ConnectPort	=> '80',
        SocksDebug	=> 0,
	Timeout 	=> 4,
	SocksVersion 	=> 5
) or die "[-] $SOCKS_ERROR";

print $sock "GET / HTTP 1.1\r\n\r\n";
print join ('',<$sock>)

Of course, you'd have to send raw http requests for things like "GET /path/to/page" etc.

NevemTeve · 05-11-2018, 02:13 PM

I think you could use 'runsocks', 'tsocks' or 'redsocks' or similar program to redict traffic to SOCKS-proxy.

keefaz · 05-13-2018, 06:58 PM

Maybe you could use threads

Code:

#!/usr/bin/perl

use strict;
use warnings;
use threads;
use Thread::Queue;

my $queue;

# worker thread
sub getUrl {
    require LWP::UserAgent;

    my $ua = LWP::UserAgent->new(
        agent => q{Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; YPC 3.2.0; .NETCLR 1.1.4322)},
    );
    $ua->proxy([qw/ http https /] => 'socks://localhost:9050'); # Tor proxy
    
    while (my $url = $queue->dequeue) {
        my $rsp = $ua->get($url);
        print $rsp->content;
    }
}

$queue = Thread::Queue->new;
my @workers;

# run 4 threads
push @workers, threads->new(\&getUrl) for (1..4);

$queue->enqueue(
    "http://url-1-withallparameters",
    "http://url-2-withallparameters",
    "http://url-3-withallparameters",
    "http://url-4-withallparameters",
    "http://url-5-withallparameters",
    "http://url-6-withallparameters"
);

$queue->end;
$_->join() foreach @workers;