[Pki-users] WG: Scep enrollment with DES3 failed when using Safenet HSM

Dear dogtag team, We’ve been using dogtag CAs (the RHEL packages in server-rpm repo) together with Safenet HSMs for some years for handling SCEP requests. We’re running into an issue again which we also had in the past: when using the HSM (a requirement here), only SCEP requests using DES for the encryption can be decoded. When DES3 is used, dogtag throws an error with “could not unwrap PKCS10 blob”. With no HSM, both algorithms work. However, the DES3 requests themselves are OK: we can unpack the inner pkcs#7, and decrypt the payload using ‘cmsutil’ (pointed at the nss db of the CA instance) and read the pkcs#10 request within. So the HSM itself has no problem decrypting. We also encountered this issue in the past with RHEL6 / DogTag 9, and it is still present with RHEL7 / Dogtag 10. At that time, we were able to configure the clients to use DES to avoid the issue, but we can’t always dictate which algorithm the clients use, and DES is nevertheless very weak. It may still be related the old BZ: https://bugzilla.redhat.com/show_bug.cgi?id=825887 and be an issue with the FIPS-2 mode (which we are using) It appears to be an issue with Dogtag. If someone has a suggestion or idea, we would appreciate hearing it. Thanks in advances BR Florian Below you can find all needed parameters and config which we used. - CA is an subca - (But dont matter, because the same issue occures also on an root ca) - SCEP enrollment works with DES encryption (HSM attached) - SCEP enrollment with DES3 works when __NO__ hsm is used - SCEP requests (DES + 3DES) can be decoded when using cmsutil direct against the HSM libary. (cmsutil -d /var/lib/pki/pkit04/alias -D -i inner_pkcs7_request.p7 -o request_des3.der ) - HSM client logs can be provided if needed - For us is seems to be a problem in dogtag. #Installed packages pki-base-java-10.3.3-19.el7_3.noarch pki-ca-10.3.3-19.el7_3.noarch pki-base-10.3.3-19.el7_3.noarch pki-tools-10.3.3-19.el7_3.x86_64 pki-kra-10.3.3-19.el7_3.noarch pki-symkey-10.3.3-19.el7_3.x86_64 pki-core-debuginfo-10.3.3-19.el7_3.x86_64 pki-server-10.3.3-19.el7_3.noarch pki-javadoc-10.3.3-19.el7_3.noarch #OS Red Hat Enterprise Linux Server release 7.3 (Maipo) #Java java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3.x86_64 #SafeNet LunaClient Version : 5.4.1 Release : 2 #SCEP enrollment profile: caRouterCert.cfg (default) #SCEP config in CS.cfg ca.scep.allowedEncryptionAlgorithms=DES3,DES ca.scep.allowedHashAlgorithms=SHA1,SHA256,SHA512 ca.scep.enable=true ca.scep.encryptionAlgorithm=DES3 ca.scep.hashAlgorithm=SHA1 ca.scep.nonceSizeLimit=16 #SSECP call ./sscep enroll -u http : / / pkit04 . eb . lan . at : 8080 / ca / cgi-bin / pkiclient . exe -c pkit04-ca.crt -k local.key -r local.csr -l cert.crt -S sha1 -E 3des #debug log [03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:263:init() CRSEnrollment: init: SCEP support is enabled. [03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:264:init() CRSEnrollment: init: SCEP nickname: pkit04:caSigningCert cert-pkit04 CA [03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:265:init() CRSEnrollment: init: CA nickname: pkit04:caSigningCert cert-pkit04 CA [03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:266:init() CRSEnrollment: init: Token name: pkit04 [03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:267:init() CRSEnrollment: init: Is SCEP using CA keys: true [03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:268:init() CRSEnrollment: init: mNonceSizeLimit: 16 [03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:269:init() CRSEnrollment: init: mHashAlgorithm: SHA1 [03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:270:init() CRSEnrollment: init: mHashAlgorithmList: SHA1,SHA256,SHA512 [03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:273:init() CRSEnrollment: init: mAllowedHashAlgorithm[0]=SHA1 [03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:273:init() CRSEnrollment: init: mAllowedHashAlgorithm[1]=SHA256 [03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:273:init() CRSEnrollment: init: mAllowedHashAlgorithm[2]=SHA512 [03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:275:init() CRSEnrollment: init: mEncryptionAlgorithm: DES3 [03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:276:init() CRSEnrollment: init: mEncryptionAlgorithmList: DES3,DES [03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:279:init() CRSEnrollment: init: mAllowedEncryptionAlgorithm[0]=DES3 [03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:279:init() CRSEnrollment: init: mAllowedEncryptionAlgorithm[1]=DES [03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:285:init() CRSEnrollment: init: mProfileId=caRouterCert [03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:349:service() operation=PKIOperation [03/Oct/2017:07:35:52][http-bio-8080-exec-1]: CRSEnrollment.java:351:service() message=MIIKywYJKoZIhvcNAQcCoIIKvDCCCrgCAQExCzAJBgUrDgMCGgUAMIIFnwYJKoZI -...snip.. t3fqG6FkBAh3L1saONZJ0pfzOnnY5CZ4aJuf5ql3XA== [03/Oct/2017:07:35:53][http-bio-8080-exec-1]: CRSEnrollment.java:920:handlePKIOperation() Processing PKCSReq [03/Oct/2017:07:35:53][http-bio-8080-exec-1]: LdapBoundConnFactory.java:324:getConn() In LdapBoundConnFactory::getConn() [03/Oct/2017:07:35:53][http-bio-8080-exec-1]: LdapBoundConnFactory.java:326:getConn() masterConn is connected: true [03/Oct/2017:07:35:53][http-bio-8080-exec-1]: LdapBoundConnFactory.java:368:getConn() getConn: conn is connected true [03/Oct/2017:07:35:53][http-bio-8080-exec-1]: LdapBoundConnFactory.java:398:getConn() getConn: mNumConns now 5 [03/Oct/2017:07:35:53][http-bio-8080-exec-1]: LdapBoundConnFactory.java:444:returnConn() returnConn: mNumConns now 6 [03/Oct/2017:07:35:53][http-bio-8080-exec-1]: CRSEnrollment.java:1164:unwrapPKCS10() failed to unwrap PKCS10 org.mozilla.jss.crypto.SymmetricKey$NotExtractableException [03/Oct/2017:07:35:53][http-bio-8080-exec-1]: CRSEnrollment.java:385:service() ServletException javax.servlet.ServletException: Couldn't handle CEP request (PKCSReq) - Could not unwrap PKCS10 blob: null #Error in localhost_access log 10.10.10.10 - - [02/Oct/2017:11:09:27 +0200] "GET / ca / cgi-bin / pkiclient . exe ? operation = PKIOperation & message = MIIKzgYJKoZIhvcNAQcCoIIKvz...snip.. HTTP/1.0" 500 3071 #Error in localhost log SEVERE: Servlet.service() for servlet [caSCEP] in context with path [/ca] threw exception [Couldn't handle CEP request (PKCSReq) - Could not unwrap PKCS10 blob: null] with root cause javax.servlet.ServletException: Couldn't handle CEP request (PKCSReq) - Could not unwrap PKCS10 blob: null at com.netscape.cms.servlet.cert.scep.CRSEnrollment.service(CRSEnrollment.java:386) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:297) at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55) at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191) at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:260) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237) at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55) at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:191) at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:187) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:186) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)