Limitations When Working with a Hardware Security Module
There are some limitations to using a private key on a Hardware Security Module (HSM) as your server key. Private keys cannot be stored on an HSM that is operated in FIPS-compliant mode.
Every HSM vendor has a list of supported and unsupported operations that can be performed or not on, for example, a key, token, or card. For more information about these limitations, refer to the documentation of the Skyhigh Security partners who provide these HSM components.
For general information about FIPS-related restrictions, see Restrictions on Secure Web Gateway in FIPS-compliant Mode.
TLS Handshake Failure
Private keys are used together with certificates to make connections secure for web traffic. Web traffic is going on over secure connections, for example, under the SSL or TLS protocol.
Using TLS 1.3 as the protocol version will lead to a handshake failure when the Hardware Security Module (HSM) that is involved here is operated in compliance with the FIPS mode. This protocol version requires raw RSA padding for sign operation, which is not supported when an HSM is operated in FIPS-compliant mode.
Running the HSM in non FIPS-compliant mode will avoid this issue. If this mode is a strict requirement, you can disable the TLS 1.3 protocol version using an option of the SSL Client Context With CA settings on Secure Web Gateway.