5:15 a.m.
Hi. I am running a DogTag server as part of a FreeIPA install. I
have an issue which has persisted following some directory replication
issues, now resolved. I had initially put the slowness down to the
replication issues but now I find errors in the PKI logs and the
slowness has persisted.
Though the services are working as intended, it's very slow and in
particular the API is sitting at 100% load (user) while performing
operations.
Extracts from the logs are pasted at
https://paste.fedoraproject.org/paste/7pz2eBm6KzItXZFFoYpldQ
I'd be very grateful for any guidance as to how to investigate further. Thanks!
Mike
Name : pki-base
Version : 10.4.1
Release : 13.el7_4
Kernel: 3.10.0-693.5.2.el7.x86_64
CentOS 7.4.1708
FreeIPA v4.5, Domain Level 1.
12:32 p.m.
Hello:
After taking a quick look at your logs, it appears there is some issue with dogtag simply
reading records from the ldap db at a pretty low level. Is there a chance the db became
corrupted at some point or something? Or was this a brand new install of the dogtag
server?
Sorry could not be more help but these symptoms are something I have not seen before.
----- Original Message -----
12:55 p.m.
Hi. Thanks for taking the time to look and reply.
There were some issues - a buildup of many cert requests had resulted
in thousands of certs being issued to two machines. I'm not sure why
this happened. As this was stopping replication, it required the
deletion of the giant entries that resulted, so it's possible that
something is inconsistent in there somewhere.
Do you think there's a way to find out which data is causing these
errors to be logged? I notice the IPA API is particularly slow. I
can probably find which entries were deleted if I get back to the
office tomorrow.
Thanks
Mike
On 17 November 2017 at 18:32, John Magne <jmagne(a)redhat.com> wrote:
12:56 p.m.
Yes, try to match the time stamps like for example the pki-tomcat ca debug
log entry
[17/Nov/2017:10:16:20][http-bio-8080-exec-2]: getEntries: exception
java.lang.ClassCastException
to the LDAP server errors log file, path similar
to /var/log/dirsrv/slapd-EXAMPLE-COM/errors
Thanks,
M.
On Fri, Nov 17, 2017 at 10:32 AM, John Magne <jmagne(a)redhat.com> wrote:
5:19 p.m.
Dogtag by itself can handle much more, by several magnitudes, but it is a
matter of sizing and tuning in various places.
May be several possible actions: revoke the non needed certs, eventually
manually delete carefully selected LDAP entries, or keep all and try tuning
one knob to see if perf is better
This will not remove the error "getEntries: exception
java.lang.ClassCastException", till need a review of the LDAP server
errrors log file for this, and eventually the
CA's /var/log/pki/pki-tomcat/ca/system
In the provided CA debug log, the embedded CA goes 2000 entries a a time in
the LDAP server, this does match the 41 times "DBVirtualList: entries:
2000" , for 79958 + 1229 = 81187 cert entries.
In the file
/etc/pki/pki-tomcat/ca/CS.cfg
check the parameter
ca.crl.pageSize
is it with a value 2000 ?
( Jack, is it the same ca.crl.pageSize used
in ./base/server/cmscore/src/com/netscape/cmscore/dbs/DBVirtualList.java ? )
Thanks,
M.
On Fri, Nov 17, 2017 at 12:58 PM, Mike Johnson <m.d.johnson(a)kuub.org> wrote:
5:36 p.m.
The ca.crl.pageSize entry is 100, which is unchanged from the default
set up by the IPA installer.
I think what I'm going to do is list the entries to delete, grab a
backup and VMware snapshot, then try the revoke/delete approach.
Worst case is I have to roll it back, and it's a small environment so
a few minutes offline to give that a go is unlikely to hurt. It's
only used 9-5 and our product uses other AAA/cert management tools to
do its stuff.
Thanks for the helpful responses!
Mike
On 17 November 2017 at 23:19, Marc Sauton <msauton(a)redhat.com> wrote:
6:09 p.m.
Removing the excess certificate entries seems to have resolved the
slowness, and I'm not seeing the
ClassCastException messages anymore. I've carried out a few
operations successfully.
The entries I removed were those like:
'cn=81178,ou=certificateRepository,ou=ca,o=ipaca' where
I had listed certs with a subject matching CN=fqdn,* and grepped out the DNs.
I should note that the machines were de-enrolled first which I suspect
reduces the risk of inconsistency?
I suspect that although Dogtag can handle many entries, the FreeIPA
web interface or API cannot.
Will monitor over the next 24 hours and roll back if necessary. Many
thanks for your helpful advice.
Mike
On 17 November 2017 at 23:36, Mike Johnson <m.d.johnson(a)kuub.org> wrote:
2745
days inactive
2746
days old
7 comments
3 participants
participants (3)
-
John Magne -
Marc Sauton -
Mike Johnson