Exchange 2010 SP3 Hub Transport Upgrade Error

Ran into a new issue yesterday related to installing Exchange 2010 SP3. I was called in to help when the initial upgrade attempt failed. The error during SP3 installation was:

An unexpected error occured while modifying the forms authentication settings for path /LM/W3SVC/1. The error returned was 5506.

A screenshot of the error is below:

Doing a search didn't come up with much, but it did give this:

• https://social.technet.microsoft.com/Forums/exchange/en-US/32df8c03-1957-4e8c-8381-4dc4cf26aee9/error-when-installing-exchange-server-2010?forum=exchangesvrgenerallegacy

That link seemed to indicate that it could be related to the SSL binding on the default web site in IIS. Taking a look at the SSL binding, it seemed to be missing the certificate assignment. However, when I tried to add the certificate I got a strange error about the session be closed.

Ok then, since you won't let me add the SSL certificate to the binding by using IIS Manager, let's try with Exchange Admin Console. When I assigned the IIS service to the certificate in EAC, it all looked fine. I also took this moment to review the certificate and verify that the SAN names were correct. I also noted that it did indicate that there was a private key for the certificate.

After this the binding worked because we could access https://servername/owa URL, but it returned a 503 error. However, rather than attempting to fix that error, we tried the SP3 install again. Since a service pack upgrade rewrites a lot of the content in the IIS virtual directories we through we might get lucky and it would fix and configuration errors that we had.

During the next install, the installation of the Hub Transport role completed successfully, but now we got an error on the Client Access role installation. As we were actively troubleshooting I didn't write it down at the time, but it was something like:

Could not grant Network Service access to the certificate with thumbprint BIGLONGHEXTHUMBPRINT because...

Based on this I decided to review the certificate in the Certificates MMC snap-in. Again, all the details looked right. Maybe I can add the necessary permissions myself for Network Service. To access the permissions for a certificate, you right-click it, point to All Tasks, and click Manage Private Keys. This normally brings up a security dialog box. However for me it brought up the following error:

Object not found.

I interpreted this error to mean that either the private keys were not really present for the certificate. Or the Domain Admin account that we were using to access the certificate and run the install didn't have permission to access the private keys. In either case, since our Domain Admin account couldn't set permissions in the certificate, we were dead in the water.

Fortunately certificates are much less expensive than they used to be and we quickly obtained a new certificate with all of the necessary names from NameCheap. They might not have the best management tools for certificates, but the price is right. So, if this didn't work it didn't waste a lot of money.

After installing the new certificate and assigning the correct services to it, we ran the Exchange 2010 SP3 upgrade again. And after some nervous waiting, the upgrade completed properly. And the upgrade fixed all of the errors for the web services. Email for phones began to work immediately, as did OWA.