PHP ldap_search size limit exceeded

Question

I'm quite new to querying Microsoft's Active Directory and encountering some difficulties:

The AD has a size limit of 1000 elements per request. I cannot change the size limit. PHP does not seem to support paging (I'm using version 5.2 and there's no way of updating the production server.)

I've so far encountered two possible solutions:

1. Sort the entries by objectSid and use filters to get all the objects. Sample Code
   I don't like that for several reasons:
   • It seems unpredictable to mess with the objectSid, as you have to take it apart, convert it to decimal, convert it back ...
   • I don't see how you can compare these id's.
     (I've tried: '&((objectClass=user)(objectSid>=0))')
     
     
2. Filter after the first letters of the object names (as suggested here):
   That's not an optimal solution as many of the users/groups in our system are prefixed with the same few letters.

So my question:

What approach is best used here?
If it's the first one, how can I be sure to handle the objectSid correctly?

Any other possibilities? Am I missing something obvious?

Update:
- This related question provides information about why the Simple Paged Results extension does not work.
- The web server is running on a Linux server, so COM objects/adoDB are not an option.

Answer 1

I was able to get around the size limitation using ldap_control_paged_result

ldap_control_paged_result is used to Enable LDAP pagination by sending the pagination control. The below function worked perfectly in my case. This would work for (PHP 5 >= 5.4.0, PHP 7)

function retrieves_users($conn)
    {
        $dn        = 'ou=,dc=,dc=';
        $filter    = "(&(objectClass=user)(objectCategory=person)(sn=*))";
        $justthese = array();

        // enable pagination with a page size of 100.
        $pageSize = 100;

        $cookie = '';

        do {
            ldap_control_paged_result($conn, $pageSize, true, $cookie);

            $result  = ldap_search($conn, $dn, $filter, $justthese);
            $entries = ldap_get_entries($conn, $result);

            if(!empty($entries)){
                for ($i = 0; $i < $entries["count"]; $i++) {
                    $data['usersLdap'][] = array(
                            'name' => $entries[$i]["cn"][0],
                            'username' => $entries[$i]["userprincipalname"][0]
                    );
                }
            }
            ldap_control_paged_result_response($conn, $result, $cookie);

        } while($cookie !== null && $cookie != '');

        return $data;
    }

If you have successfully updated your server by now, then the function above can get all the entries. I am using this function to get all users in our AD.

Answer 2

As I've not found any clean solutions I decided to go with the first approach: Filtering By Object-Sids.

This workaround has it's limitations:

1. It only works for objects with an objectsid, i.e Users and Groups.
2. It assumes that all Users/Groups are created by the same authority.
3. It assumes that there are not more missing relative SIDs than the size limit.

The idea is it to first read all possible objects and pick out the one with the lowest relative SID. The relative SID is the last chunk in the SID:

S-1-5-21-3188256696-111411151-3922474875-1158

Let's assume this is the lowest relative SID in a search that only returned 'Partial Search Results'. Let's further assume the size limit is 1000.

The program then does the following: It searches all Objects with the SIDs between

S-1-5-21-3188256696-111411151-3922474875-1158
and
S-1-5-21-3188256696-111411151-3922474875-0159

then all between

S-1-5-21-3188256696-111411151-3922474875-1158
and
S-1-5-21-3188256696-111411151-3922474875-2157

and so on until one of the searches returns zero objects.

There are several problems with this approach, but it's sufficient for my purposes.
The Code:

$filter = '(objectClass=Group)';
$attributes = array('objectsid','cn'); //objectsid needs to be set

$result = array();

$maxPageSize = 1000;
$searchStep = $maxPageSize-1;

$adResult = @$adConn->search($filter,$attributes); //Supress warning for first query (because it exceeds the size limit)

//Read smallest RID from the resultset
$minGroupRID = '';

for($i=0;$i<$adResult['count'];$i++){
    $groupRID = unpack('V',substr($adResult[$i]['objectsid'][0],24));
    if($minGroupRID == '' || $minGroupRID>$groupRID[1]){
        $minGroupRID = $groupRID[1];
    }    
}

$sidPrefix =  substr($adResult[$i-1]['objectsid'][0],0,24);   //Read last objectsid and cut off the prefix

$nextStepGroupRID = $minGroupRID;

do{ //Search for all objects with a lower objectsid than minGroupRID
    $adResult = $adConn->search('(&'.$filter.'(objectsid<='.preg_replace('/../','\\\\$0',bin2hex($sidPrefix.pack('V',$nextStepGroupRID))).')(objectsid>='.preg_replace('/../','\\\\$0',bin2hex($sidPrefix.pack('V',$nextStepGroupRID-$searchStep))).'))', $attributes);

    for($i=0;$i<$adResult['count'];$i++){
        $RID = unpack('V',substr($adResult[$i]['objectsid'][0],24));    //Extract the relative SID from the SID
        $RIDs[] = $RID[1];

        $resultSet = array();
        foreach($attributes as $attribute){
            $resultSet[$attribute] = $adResult[$i][$attribute][0];
        }
        $result[$RID[1]] = $resultSet;
    }

    $nextStepGroupRID = $nextStepGroupRID-$searchStep;

}while($adResult['count']>1);

$nextStepGroupRID = $minGroupRID;

do{ //Search for all object with a higher objectsid than minGroupRID  
    $adResult = $adConn->search('(&'.$filter.'(objectsid>='.preg_replace('/../','\\\\$0',bin2hex($sidPrefix.pack('V',$nextStepGroupRID))).')(objectsid<='.preg_replace('/../','\\\\$0',bin2hex($sidPrefix.pack('V',$nextStepGroupRID+$searchStep))).'))', $attributes);

    for($i=0;$i<$adResult['count'];$i++){
        $RID = unpack('V',substr($adResult[$i]['objectsid'][0],24));    //Extract the relative SID from the SID
        $RIDs[] = $RID[1];

        $resultSet = array();
        foreach($attributes as $attribute){
            $resultSet[$attribute] = $adResult[$i][$attribute][0];
        }
        $result[$RID[1]] = $resultSet;
    }

    $nextStepGroupRID = $nextStepGroupRID+$searchStep;

}while($adResult['count']>1);

var_dump($result);

The $adConn->search method looks like this:

function search($filter, $attributes = false, $base_dn = null) {
        if(!isset($base_dn)){
            $base_dn = $this->baseDN;
        }

        $entries = false;
        if (is_string($filter) && $this->bind) {
                if (is_array($attributes)) {
                        $search  = ldap_search($this->resource, $base_dn, $filter, $attributes);
                } else {
                        $search  = ldap_search($this->resource, $base_dn, $filter);
                }
                if ($search !== false) {
                        $entries = ldap_get_entries($this->resource, $search);
                }
        }
        return $entries;
}

Answer 3

Never make assumptions about servers or server configuration, this leads to brittle code and unexpected, sometimes spectacular failures. Just because it is AD today does not mean it will be tomorrow, or that Microsoft will not change the default limit in the server. I recently dealt with a situation where client code was written with the tribal knowledge that the size limit was 2000, and when administrators, for reasons of their own, changed the size limit, the client code failed horribly.

Are you sure that PHP does not support request controls (the simple paged result extension is a request control)? I wrote an article about "LDAP: Simple Paged Results", and though the article sample code is Java, the concepts are important, not the language. See also "LDAP: Programming Practices".