[tech] OpenLDAP olcSizeLimit size limits exceeded

[Nick Bannon]

On Fri, Feb 20, 2015 at 02:45:16PM +0800, David Adam wrote:
> Bumped up again... need a proper fix.
> 
> On Mon, 28 Jul 2014, David Adam wrote:
> > Tonight, [ASH] was trying to add a new user. The membername.ucc.asn.au 
> > alias wasn't being created correctly, and it turned out this was because 
> > the "list all users" operation in Zonemake (analagous to `getent passwd`) 
> > was hitting the maximum number of answers to a query in the LDAP server 
> > (1000).
> > 
> > I've bumped up the maxium size to 1500 (olcSizeLimit: 1500), but perhaps 
> > we could consider:
> > 
> > a) retiring some old usernames. `getent passwd | grep locked2004` might be 
> > a good place to start; 10 years to reactivate should be enough.
> > b) enabling paged queries (setting `pagesize 500` or so in nslcd.conf on 
> > all our clients, and setting `olcSizeLimit: 1500 size.prtotal=unlimited` 
> > or similar
> > c) removing size limits on the server altogether; I don't think we're ever 
> > going to have enough entries to seriously cause performance degradation.

I think I got it to stick. Previous config:
mussel:/# grep -R -i olcSizeLimit: /etc/ldap/slapd.d
/etc/ldap/slapd.d/cn=config.ldif:olcSizeLimit: unlimited
/etc/ldap/slapd.d/cn=config/olcDatabase={-1}frontend.ldif:olcSizeLimit: 1000

New config:
/etc/ldap/slapd.d/cn=config.ldif:olcSizeLimit: unlimited
/etc/ldap/slapd.d/cn=config/olcDatabase={-1}frontend.ldif:olcSizeLimit: unlimited

Both settings are necessary, if they're missing it defaults to 500
results.

Changed with:
mussel:/etc/ldap/slapd.d# echo "
dn: cn=config
changetype: modify
replace: olcSizeLimit
olcSizeLimit: unlimited

dn: olcDatabase={-1}frontend,cn=config
changetype: modify
replace: olcSizeLimit
olcSizeLimit: unlimited
"|ldapmodify -H ldapi:///

Part of the strangeness was:
  * we're using On-Line Configuration (OLC) in /etc/ldap/slap.d now,
    not any literal slapd.conf file, this scatters the config in
    hard-to-type directory/filenames
  * changing the first cn=config fixed it temporarily, but
    olcDatabase={-1}frontend,cn=config brought the sizelimit back
    on restart.
  * the OpenLDAP slapd.conf(5) documented default is 500 results, but
    1000 was coming from somewhere
  * it's actually a hard maximum results limit, not a per-page limit
  * ...which makes paged mode or setting /etc/nslcd.conf:pagesize seem
    like a red herring. Clients can use paged mode if they like, but
    there's no point in the server capping the total results.

Setting "SizeLimit: unlimited" or "SizeLimit: size.prtotal=unlimited"
(500 results per page, unlimited pages; if the ldapsearch or nslcd
clients use paging) on mussel temporarily fixed it:
	mussel:/etc/ldap/slapd.d# printf "version: 1\n\ndn: cn=config\nchangetype: modify\nreplace: olcSizeLimit\nolcSizeLimit: unlimited\n-\n" | ldapmodify -H ldapi:///

(or "ldapmodify -H ldapi:/// -Y EXTERNAL -D cn=config")

Setting it on motsugo failed, because it's the slave, not the master:
ldap_modify: Server is unwilling to perform (53)
        additional info: shadow context; no update referral

but on motsugo, "systemctl restart slapd.service" broke the results
coming from motsugo.
motsugo# ldapsearch -H ldapi:/// -Y EXTERNAL -E pr=600/noprompt |grep '^# num'
# numResponses: 1002
# numEntries: 1000

...until one fixed it on mussel again.

"slapcat -H ldap:///cn=config" helped figure out what ldapmodify dn= I
needed to change the frontend config.

Nick.

-- 
   Nick Bannon   | "I made this letter longer than usual because
nick-sig at rcpt.to | I lack the time to make it shorter." - Pascal