Ever wonder why documents always say use module protect when using FIM CM and Thales HSM together? Why use the HSM in a less secure mode when it is designed to be a K of N device?
Recently, I had a customer that wanted to use the highest possible security in their PKI and smart card implementation. They outsourced their IT and wanted to have all the PKI components controlled by one employee and one contractor. This is best practice in this environment as it keeps the system secure by limiting access.
The customer wanted a 2 of 6 implementation, taking 2 of the 6 smart cards programmed for the device to authenticate to the device. This worked well for the CA components and the customer then wanted the FIM CM servers setup the same way. I found no documentation or articles that said the K of N would not work for FIM CM.
The server setup was done with the K of N for FIM CM and the install and setup went well…or so I thought. When trying to issue a smart card from FIM, the system stalled on initializing the smart card.
At this point, the FIM CM system should have been asking for the passphrase and OCS cards. Nothing was showing up on the server desktop or in session 0. Now the real fun begins.
After looking through event logs and any other debug logs available, there did not appear to be any errors. We let the system run for 24 hours on the initializing card dialog thinking that maybe the system was just slow. No luck. No error messages and no indication of an issue. So what is the next step?
We decided to open a request for assistance from Microsoft. After going through several days of obtaining event and debug logs, the tech stated it was not a Microsoft problem, but a Thales problem. So off to Thales support.
Thales asked for more event and debug logs. After about a week, they declared it was not a Thales issue, but a Microsoft issue. Back to square one.
Being a developer of crypto service providers in a past life, I decided to dig deeper into the problem since no one at Thales or Microsoft seemed to be able to help. Microsoft never tested FIM CM with any type of HSM and Thales never tested with FIM CM.
After installing several debug components and using a kernel debugger, I found several discrepancies in both the Microsoft implementation of FIM CM and of the Thales implementation of the CSP used by FIM CM for access to the HSM.
After a few days of digging I found out why K of N has no chance of working with FIM CM and any HSM. Microsoft attempts to open the session to the HSM with a CryptAcquireContext call with the crypt_silent flag set. This tells the CSP not to request the passphrases for the OCS cards. Light blub number one went on.
Then I found that the Thales CSP returns a success to the CryptAcquireContext when accessing the HSM. Going back to the Microsoft specifications for CSP development, it says that during the CryptAcquireContext if a passphrase is needed and the crypt_silent flag is requested, then an error code should be returned. Thales did not comply with the specifications, since at the time of the CryptAcquireContext they do not know a passphrase is required.
FIM CM will hang when the next crypto call is made to the HSM. Microsoft should at least timeout after a period of time and Thales should correct the CSP.
With Module protect, no passphrase is required and the system works.
Now you know why K of N usage of an HSM will not work with FIM CM.