I came upon a blog post on Scott Lowe’s blog suggesting a solution to resolve AD integration issues where more than 1,000 results are returned in a query on some UNIX/LINUX systems.  I will try to explain why this is a less than optimal solution, which could cause performance issues with the directory server.

What is MaxPageSize, and how does it affect me?

The problem at hand is, that certain LDAP clients can only receive the first 1,000 results when they make an LDAP query to Active Directory, when more results exist that satisfy the query scope.  If there are 1,200 results to be returned, the client will receive the "Size Limit Exceeded" error when the paging control is not used.

The default LDAP Policy settings for Active Directory limit the amount of results returned in a single page to 1,000 results.  So clients that do not implement paging support, will only be able to retrieve the first page of 1,000 results.

Microsoft lists MaxPageSize as:

MaxPageSize – This value controls the maximum number of objects that are returned in a single search result, independent of how large each returned object is. To perform a search where the result might exceed this number of objects, the client must specify the paged search control. This is to group the returned results in groups that are no larger than the MaxPageSize value. To summarize, MaxPageSize controls the number of objects that are returned in a single search result.
Default value: 1,000

What is Paging Support?

LDAP was designed with extensibility in mind, allowing for controls and extensions to be implemented to enhance the functionality of the directory.  Active Directory (ADDS and ADLDS) is a LDAP v3 compliant directory, which implements the LDAP Control Extension for Simple Paged Results Manipulation (RFC2696).

This control extension allows the client to make a query to the LDAP server, inspect the first page of data, and be told the entire result set count.  The client can process the results, and request the next page of results until all results are complete.  This prevents a query which matches a large set of objects, from returning more results than the client is prepared to handle, and also protects the server from returning large result sets as a single operation.

Microsoft also lists these as benefits of using paging:

  • Lower network traffic
  • The client can be more responsive to the user because the client returns only one page of data rather than all of the data in the result set.
  • Fewer resources are needed by the client and server
  • The client can abandon the search before completion

I suppose a good way to explain it is, a throttle control the client has in retrieving the results that satisfied the query.  The client processes the first page of results, then makes a request for the next page from the server when the client is ready.

Should I change the Default Value to a higher value?

A common request from application owners which either do not enable paging, or are using an LDAP library which does not implement paging, is why can’t we just increase the MaxPageSize to a higher number? 

My first reaction is,  if 1,000 is not enough, what is the right number it should be?  1,001 or maybe 5,000?  What if you still have 5,001 results in your query?

For example, IBM recommends changing the MaxPageSize to 100,000 since their Lotus Sametime product does not support paging.  This would pretty much nullify paging ability except in the largest of directories, since most queries would not return a result set this large.  Yet, what if 100,001 results existed that you wanted to retrieve without receiving the "Size Limit Exceeded" error?

Unfortunately many applications seem to think it’s a good idea to alter the entire infrastructure because the application does not support paging, which I feel is a bad design concept.

For some queries, 2,000 is enough, while others it is still not large enough to return all the satisfying objects.

It becomes a moving target as application needs and populations change within the directory.  If you change it for one application, how do you satisfy all applications without adversely affecting the server?

Query policies are effective for the entire server, so any every LDAP query will return results with the maxPageSize.  It is not on a per user, or client server basis, but can be applied at the Forest, Site or Domain Controller level.

By increasing the maxPageSize to satisfy the application, you risk the chance of overloading the server which could yield a self induced denial of service as the server responds to queries.

So if I shouldn’t increase MaxPageSize, what can I do to get all the results?

I hope I have swayed some readers from increasing the default value, but unfortunately this does not resolve the application issue at hand.

So instead of changing the value here are some suggestions to attempt to return the entire results for the application.

  1. Implement Paging Support in the application – This seems like the most obvious, as if you can utilize paging on the application client you will be able to return the complete result set without adversely affecting the AD server.  I tried to provide links below which might help application developers.  It might be possible that the application already supports paging, but it is not enabled by default, so consult your application developer as well.
  2. Refine your Query to return less results – Evaluate the query attempted and see if it can be refined to return a smaller, more exact, set of results which may be under the MaxPageSize.  Instead of looking for all the "John Smith"’s,  maybe also ask for the John Smith’s in a specific OU subtree, or with more specific attributes like location or country.  This is not always possible especially in large directories where the intention is to return a broad scope of objects.
  3. Isolate your problem applications – The LDAP query policies are defined on the Forest, Site, and Domain Controller scope.  So you can specify an LDAP query policy in an isolated site or specific server you have setup just for the applications which do not utilize paging.  You can increase the MaxPageSize values for these specific instances, without affecting all of your instances. This does not insulate the server from having performance issues, but it can isolate other applications and NOS services from being affected.  If possible also remove it’s SRV records so other clients cannot discover it for other services.

If you can think of any other potential work around’s when paging is not supported on the application, please let me know within the comments.

 

Further Reading

Limiting LDAP searches with MaxPageSize

Naming and Directory (JNDI) – JNDI, Active Directory, Paging and Range Retrieval

JNDI, Active Directory and LDAP Extended Controls (LDAP Stats, Verify Name)  

Bug in Microsoft LDAP API

NSS_LDAP C Library which supports Paging

Paging in System.DirectoryServices.Protocols

Creating more efficient Active Directory-Enabled Applications

 

Technorati Tags: ,