After happily running for several years, one of the Skype for Business edge servers for one implementation decided it was not going to start its Audio/Video Authentication and Audio/Video Edge service!
Looking at the event viewer, the following two Event IDs were raised: 19008 and 19005. Specifically: 19008: Private key for server certificate not found by the LS A/V Authentication service or the service does not have sufficient permissions to access the certificate.
After verifying the private key permissions are set correctly (NETWORK SERVICE: Read, etc) in the Certificate MMC snap-in, I checked to see what the certs looked like in PowerShell
PS Cert:\LocalMachine\My\> dir .\ 5E670E493E5EBAACC5B26E219ACA8A629F9485D4 | flHasPrivateKey : True PrivateKey : PublicKey : System.Security.Cryptography.X509Certificates.PublicKey
Notice that there is no PrivateKey provider defined here, which means the cert broke somehow! Strange, as in this environment there were two Skype for Business edge servers, one worked perfectly, the other did not.
Anyways, the fix was to tear the certs apart, and put them back together as shown in this Merge certificate public and private key with OpenSSL TechNet article.
Specifically
-
- I got the OpenSSL binaries from: https://indy.fulgan.com/SSL
- I extracted the keys using the following commands:
openssl pkcs12 -in egdev1.pfx -nocerts -out private_key.pem -nodes openssl pkcs12 -in egdev1.pfx -nokeys -out public_key.crt - I merged the keys back together using the following command:
openssl pkcs12 -export -in public_key.crt -inkey private_key.pem -out lync_edge_merged.pfx
After certificate import, and applying it to the services, I checked to see what the certs looked like in PowerShell
PS Cert:\LocalMachine\My\> dir .\ 5E670E493E5EBAACC5B26E219ACA8A629F9485D4 | flHasPrivateKey : True PrivateKey : System.Security.Cryptography.RSACryptoServiceProvider PublicKey : System.Security.Cryptography.X509Certificates.PublicKey
Notice that there is now a PrivateKey provider defined here, and the two Audio/Video Authentication and Audio/Video Edge services started up just perfectly!
Fantastic!
I had the exact same problem. Some articles suggested assigning the internal certificate to the AV service but that just didn’t feel right since we’ve had the public cert assigned for four years without a problem. The problem occurred when updating to a new public cert. This article solved it. Thanks!
Love it! Glad this solved your problem!
Thanks! You saved my day!
You’re more than welcome!
I don’t normally comment on these, but I want to leave some breadcrumbs in Google for people who may run in to similar issues that I have. We have spent the last few weeks going through a round of 3 year cert expirations and replacements. Every public cert we get from Digicert (CSR generated in Windows, cert issued from Digicert) has come with this EXACT situation: Private Key provider blank when looking at the cert in powershell. I found this page because we had this exact issue with Skype when we rolled those certs. However, we also had issues with other MS platforms as well:
Dynamics / CRM: After installing a new cert where the Private Key was blank instead of showing the provider, CRM would not allow people to log in. The metadata file for integration with ADFS existed and was accessible, but the file was significantly shorter than it should have been and included errors indicating that it could not correctly generate the metadata.
Exchange 2013 – OWA and ECP “login loop”: Using a cert with this issue on Exchange results in strange behavior as well. Email sends and arrives without issue. OWA and ECP pages will load and present the cert as valid for TLS. If you are using Basic Auth or Windows Auth (get-ecpvirtualdirectory | fl *, get-owavirtualdirectory | fl *), both services will work without fail. However, if you are using Forms Auth for OWA and ECP then logins will fail. The page will load normally but when you give valid creds it will dump you back to the login page immediately. There are several docs out there on Exchange OWA/ECP “login loops”, but I did not find any that addressed this issue.
All of these issues were fixed by the OpenSSL steps Luke has outlined here.
What I am unsure of is why I have recently gotten so many certs from Digicert that display this issue. I know there was a recent RFC that made changes to the issuance of publicly trusted certs (underscores are no longer allowed, for example) but I have not sat down and read all the content of the RFC to see if this behavior change was a part of it. Perhaps it is something about CSRs coming from Server 2016.
Anywho, I hope this helps someone the way it helped us. MS products seem to depend on seeing the PrivateKey provider for some strange, specific uses.
Thanks!
Thanks for the comment Mike, happy to have helped!
Thank you so much for posting this fix also! Work for me after wasting 2 hours trying to figure out why the service would not start with the new cert.
You’re very welcome! I actually come back to this article often, as I seem to come across this scenario every now and then.
This fix saved my life. Thank you very much! Now please tell me why pfx exported WITH private key from one server doesn’t work on another server? And it works perfectly well after splitting and merging with openssl?
I haven’t actually looked into it yet, but great question, I will one of these days. 🙂
I had exactly this problem but my certificate had been created using OpenSSL although I saw exactly the same missing provider. I simply deleted my certificate which I’d originally imported using Import-PfxCertificate and imported it again using the Certificate Manager in MMC.
I had exactly this problem but interestingly my certificate had been created using OpenSSL.
I had originally imported it via Import-PfxCertificate and saw exactly the same missing PrivateKey Provider. Using MMC’s Certificate Manager I deleted it and then imported exactly the same file again and it worked.
Yes, that is always step 1 in troubleshooting the process. Sometimes it works, and sometimes a rebuild is necessary. – I’ve used those two processes so many times I’ve lost count.