Login to PopAds.net

Question

I've searched this website and google but just can't seem to figure out how to get this to work. I'm trying to login to popads.net using a PHP script so I can compile my earnings for my websites onto one page. But this website is giving me trouble. Can anyone see what I'm doing wrong?

<?php
//username and password of account
$username = 'myusername';
$password = 'mypassword';

//set the directory for the cookie using defined document root var
$path = DOC_ROOT."/ctemp";

//login form action url
$url="https://www.popads.net/users/login"; 
$postinfo = "data[User][username]=".$username."&data[User][password]=".$password;

$cookie_file_path = $path."/cookie.txt";

$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_NOBODY, false);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);

curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
//set the cookie the site has for certain features, this is optional
curl_setopt($ch, CURLOPT_COOKIE, "cookiename=0");
curl_setopt($ch, CURLOPT_USERAGENT,
    "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.12) Gecko/20050915 Firefox/1.0.7");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_REFERER, $_SERVER['REQUEST_URI']);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);

curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postinfo);
curl_exec($ch);

//page with the content I want to grab
curl_setopt($ch, CURLOPT_URL, "https://www.popads.net/users/dashboard");
//do stuff with the info with DomDocument() etc
$html = curl_exec($ch);
echo $html;
curl_close($ch);

Answer 1

As this site is using CSRF protection, in order to login you need to first get the CSRF token from the original form and pass this with the login data to the login endpoint. The CSRF token has the field name data[_Token][key] on the homepage. The site may also set a cookie when it sets this so you will need to pass that cookie data back if you get it from cURL.

This said: my recommendation is to see if they have an official API, and before coding a scraper yourself, make sure you're not breeching any terms and conditions which could get you blacklisted.

Answer 2

EDIT 03/26/2017

At some point, PopAds switched to using client side Javascript to generate a check value using two server side variables retrieved from an AJAX request. It looks easy enough to replicate in PHP but since the JS can be easily changed, let's not play cat and mouse and just use an engine to handle the JS for us.

Here is some PHP code to run a CasperJS script to log in and get what we need. First, you'll need to install phpcasperjs/phpcasperjs using Composer. You'll also need nodejs installed, and install the following modules to the directory where you plan to run this script: npm install phantomjs ; npm install casperjs

<?php

require_once 'vendor/autoload.php';

use Browser\Casper;

define('POPADS_EMAIL', '[email protected]');
define('POPADS_PASS',  'your password');

$casper = new Casper(__DIR__ . '/node_modules/casperjs/bin/');

//$casper->setOptions(['engine' => 'slimerjs']);
//$casper->setDebug(true);

// change the UA!
$casper->setUserAgent('Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0');

// navigate to google web page
$casper->start('https://www.popads.net/');

// wait for text if needed for 3 seconds
$casper->waitForText('Reset password', 5000);

//data[User][username]
//data[User][password]


$casper->fillFormSelectors(
    'form.didEnabled',
    array(
        'input#UserUsername' => POPADS_EMAIL,
        'input#UserPassword' => POPADS_PASS,
    ),
    true
);

$casper->waitForText('</body>', 5000);

$casper->capturePage(__DIR__ . '/login.jpg');

// run the casper script
$casper->run();

// need to debug? just check the casper output
//$output = $casper->getOutput();

$output = $casper->getHTML();

if (strpos($output, 'PopAds - Dashboard') !== false) {
    echo "Logged in!";  
} else {
    echo "Login failed.";

    var_dump($output);
}

Here is a working example. I added some notes to the code. This no longer works, left for reference.

The basic process is:

• Request the login form page
• Extract "sid" token value from login form
• Extract form inputs
• Populate form inputs with your login info
• Send login post request
• Extract balance from the resulting page after being logged in

And the code:

<?php

error_reporting(E_ALL);ini_set('display_errors', 1);

// credentials
$USERNAME = 'username';
$PASSWORD = 'password';

// login url
$LOGINURL = 'https://www.popads.net/users/login';

// initialize curl
$ch = curl_init();
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_COOKIEFILE, ''); // empty file means curl will keep cookies for the lifetime of the handle
// use cookiejar if you'd like to save the cookies for more than the request
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 120);
curl_setopt($ch, CURLOPT_TIMEOUT, 120);

// set URL and request (establishes cookies, gets login sid)
curl_setopt($ch, CURLOPT_URL, $LOGINURL);
$data = curl_exec($ch);

// look for "sid" value on form action (required)
preg_match('#/users/login\?sid=([\w\d]+)#', $data, $match);
$sid        = $match[1];

// extract form fields from form
$formFields = getFormFields($data);

// set username and password
$formFields['data[User][username]'] = $USERNAME;
$formFields['data[User][password]'] = $PASSWORD;

// build http post string
$post_string = http_build_query($formFields);

// update login url with sid value and post login form
curl_setopt($ch, CURLOPT_URL, $LOGINURL . '?sid=' . $sid);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_string);

// execute login request (should be logged in at this point)
$result = curl_exec($ch);

// get balance from page
preg_match('#<h5>Current Balance:</h5>\s*<div class="overview overview_green">(.*?)</div>#is', $result, $balance);
$balance = trim($balance[1]);

// show balance
echo "Your balance is $balance<br>";


function getFormFields($data)
{
    if (preg_match('/(<form.*?id=.?UserLoginForm.*?<\/form>)/is', $data, $matches)) {
        $inputs = getInputs($matches[1]);

        return $inputs;
    } else {
        die('didnt find login form');
    }
}

function getInputs($form)
{
    $inputs = array();

    $elements = preg_match_all('/(<input[^>]+>)/is', $form, $matches);

    if ($elements > 0) {
        for($i = 0; $i < $elements; $i++) {
            $el = preg_replace('/\s{2,}/', ' ', $matches[1][$i]);

            if (preg_match('/name=(?:["\'])?([^"\'\s]*)/i', $el, $name)) {
                $name  = $name[1];
                $value = '';

                if (preg_match('/value=(?:["\'])?([^"\'\s]*)/i', $el, $value)) {
                    $value = $value[1];
                }

                $inputs[$name] = $value;
            }
        }
    }

    return $inputs;
}

Output:

Your balance is $0.00

Answer 3

Add:

# Send previously received cookies.
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path);

Setting CURLOPT_COOKIEJAR only saves cookies received.

Delete:

curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");

This duplicates the next line where you set CURLOPT_POST.

Finally, the second page you are downloading (Dashboard) needs to be fetched with regular GET method. Add this before the second curl_exec() call:

curl_setopt($ch, CURLOPT_HTTPGET, true);