Showing posts with label Exchange. Show all posts
Showing posts with label Exchange. Show all posts

Wednesday, May 30, 2018

Expired Microsoft Exchange Server Auth Certificate

When you install your first Exchange Server 2013 or Exchange Server 2016 server, a certificate with the friendly name Microsoft Exchange Server Auth Certificate is created. This certificate is self-signed and used for OAuth authentication between applications such as Exchange Server and SharePoint. However, it is also used for hybrid deployments between on-premises Exchange Server and Exchange Online.

This certificate is unique because it is installed on all of your Exchange servers. The subject for the certificate is "CN=Microsoft Exchange Server Auth Certificate" and does not contain any SAN names with references to specific servers.

It also has a 5-year lifetime. Which is just long enough for everyone to forget about it. I suspect that this certificate is due to expire in many organizations soon.


Today I got a call from an organization with the following symptoms:
  • Outlook clients were slow to start
  • Outlook clients were not displaying the user's calendar
  • Outlook clients were not able to access Out-of-Office settings
  • Everything was functioning fine the previous day
  • No system changes had been made
  • The organization is configured in hybrid mode with Exchange Online, but the issue was occurring for on-premises users
  • Accessing through OWA was unaffected
An initial review of the configuration didn't identify anything out of the ordinary except that the Microsoft Exchange Server Auth Certificate had expired about two weeks ago. There was also this related event in the Application log:


The event information is:

  Source: MSExchange Web Services
  Event ID: 24
  
  The Exchange certificate [Subject]
  CN=Microsoft Exchange Server Auth Certificate

  [Issuer]
  CN=Microsoft Exchange Server Auth Certificate

  [Serial Number]
  3329C02443B435A447B92E81EB177303

  [Not Before]
  2013-06-11 6:25:31 PM

  [Not After]
  2018-05-16 6:25:31 PM

  [Thumbprint]
  DFC06BF3FC76974BCD1C050340998B65D5491007
  expired on 2018-05-16 6:25:31 PM.

This certificate is not bound to IIS, so why is this expiration being noted by MSExchange Web Services?

The certificate used for authorization is set separately from IIS but is configurable. You can see the currently configured certificate by using Get-AuthConfig. The thumbprint for the certificate is listed in the CurrentCertificateThumbprint property.


If you've already been searching around related to this certificate, you've probably found some instructions on how to recreate it. However, if it is expired, you can just renew it instead by using the Exchange Admin Console. In servers > certificates, select Microsoft Exchange Server Auth Certificate and then click Renew in the details pane as shown below. The screen shot below is of a certificate that is not expired yet, it looks exactly the same other than the expiry date.

Renewing creates a second certificate named Microsoft Exchange Server Auth Certificate that is valid for another 5 years. This certificate has a new thumbprint and exists only on the server you've renewed it on. You need to identify the thumbprint for the new certificate. If you edit the certificate, in Exchange Admin Center, the thumbprint is on the general tab as shown below. You can type this in, but you're probably better of to cut and paste it into the later commands.





My preference is to put the required information for the next command into variable. We need the thumbprint of the new certificate and the current date.

$thumb = "NewCertificateThumbprint"
$date = get-date

Then run the following command to add the new certificate:

Set-AuthConfig -NewCertificateThumbprint $thumb -NewCertificateEffectiveDate $date

You will get a warning that the new effective date is not 48 hours in the future. However, if we're recovering from an expired certificate, we're OK with that.

Now you need to publish the certificate to all servers:

Set-AuthConfig -PublishCertificate

And finally, remove the old expired certificate from the configuration:

Set-AuthConfig -ClearPreviousCertificate


Finally, since we've removed all references to the expired certificate you can delete it from all Exchange servers.

You may find that you need to do an iisreset after all of the AuthConfig changes were done. I found some examples where they ran iisreset.exe and didn't take the time to experiment. It's possible that it will pickup without a restart but just take longer.

I've added another post related to the Auth certificate and hybrid deployments here:




Thursday, March 22, 2018

Update Source Anchor to ms-DS-ConsistencyGUID

A key part of most Office 365 deployments is directory synchronization with on-premises Active Directory (AD). To maintain a link between individual object in AD and Office 365, one attribute in AD is defined as the source anchor. The source anchor acts as a unique identifier for each object so that you can change properties like UPN and have them replicate to the proper matching object in Office 365.

Older versions of Azure AD Connect used the objectGUID attribute as the source anchor. In many cases, this is a good choice because it is an attribute that doesn't change. However, in complex configurations with multiple AD forests, this can cause an issue during migrations. When a user account is moved from one AD forest to another, it gets a new objectGUID in the new forest. Due to the new objectGUID , Azure AD Connect treats the user as a new user instead of an existing user.

To avoid this issue, we need to use an attribute for source anchor that can be moved between forests. Microsoft now recommends that deployments of Azure AD Connect use the mS-DS-ConsistencyGuid attribute as the source anchor. Unlike objectGUID , this attribute can be modified. So, it can be maintained when users are moved between forests.

I've just recently completed this process and I'll provide some general guidance (for my own reference) and a few items that I didn't see anywhere else.

Updating the Source Anchor

First, read the information from Microsoft on updating the source anchor. You'll need to scroll down a bit to the heading "How to enable the ConsistencyGuid feature - Existing deployment".
This documentation provides click by click instructions on how to use the Azure AD Connect configuration wizard and work through the Change Source Anchor option. This wizard will:
  • Require you to enter Azure AD credentials for a global admin
  • Require you to enter a user account with AD FS admin permissions (if integrated with AD FS)
  • Do a prerequisite check
  • Update necessary settings in Azure AD
  • Update necessary settings on connectors for all connected forests
  • Update AD FS claim rules
If you're like me, before attempting this you'll scour the web looking for information about how to fix things when they go wrong. Unless things change a lot after I write this, you won't find much. The reason you won't find much is because the wizard does exactly what it's supposed to.

Note: The wizard does not ask you which attribute you want for the source anchor. It is assumed that you want to use mS-DS-ConsistencyGuid.

Change Details

When the wizard is done, a full sync is triggered for all connected forests. As part of the full sync Azure AD Connect reviews the value of the mS-DS-ConsistencyGuid attribute. If the value is blank, the objectGUID value is copied into the mS-DS-ConsistencyGuid value. Then on the next sync cycle the mS-DS-ConsistencyGuid value is sent to Office 365.

If you are using AD FS, the wizard updates the AD FS claims rules for Office 365. I didn't do a detailed review of the rules, but it seems to be similar (if not the same) as the rules Microsoft documents for you to make manual changes. You can see these rules here:
The Modify AD FS Claim Rules section in the above link has four rules. The net effect of these rules is to use mS-DS-ConsistencyGuid if it is populated and use objectGUID otherwise. This avoids downtime when AD FS rules have been modified but the mS-DS-ConsistencyGuid hasn't been populated yet.

Permissions to mS-DS-ConsistencyGuid

If you have let the installation and configuration wizards for Azure AD Connect manage all of the service account permissions and AD FS integration, there should be no issues. However, if you've been manually configuring the permissions and AD FS rules, you might need to make changes manually.

The only permission change required when changing the source anchor is ensuring that the service account in each domain has read and write permissions to the mS-DS-ConsistencyGuid attribute. Unfortunately when you attempt to assign these permissions by using Active Directory Users and Computers, you find out that the attribute is not displayed in the GUI for modifying permissions. To set the permissions, use the following command (please note that this is a single line):
dsacls.exe "dc=contoso,dc=com" /I:S /G "contoso\serviceaccount:RPWP;mS-DS-ConsistencyGuid;user"

The above command grants (/G) inheritable (/I:S) permissions to the service account to read (RP) and write (WP)  the mS-DS-ConsistencyGuid attribute on user objects.

Verifying Successful Implementation

Obviously after a successful implementation, your users can still sign in to Office 365, but there are a few other things you can verify:
  • Review the AD FS claim rules for Office 365 and verify that they were modified.
  • Review the mS-DS-ConsistencyGuid attribute on several user objects to verify that it has been populated.
On one deployment, we had an error in the Azure AD Connect full sync for one domain immediately after finishing the wizard. The domain that finished properly updated the mS-DS-ConsistencyGuid attribute and the domain with the error didn't. Running an incremental sync didn't fix it. We needed to force a full sync again to get the attribute updated properly.

When viewing the value of the mS-DS-ConsistencyGuid attribute in Active Directory Users and Computers, the value might be displayed in special characters for some users. This is a display error in Active Directory Users and Computers. This is not something you need to fix.

For Further Reading


Monday, January 22, 2018

All Certificate Names MUST be in Subject Alternative Names

This has been popping up for a while, but it's worth pointing out again. When you get a SAN/UCC certificate, the DNS name that you use for the subject (common name) also needs to be in the subject alternative names attribute.

For example:
  • Subject:
    • webmail.contoso.com
  • Subject alternative names
    • webmail.contoso.com
    • exch1.contoso.com
    • exch2.contoso.com
    • Autodiscover.contoso.com
When you get your certificate from a public certificate authority (CA), most of them ensure that the subject is also added as a subject alternative name. However, some might not and you should watch for it.

It's important to note that the Microsoft CA does not automatically add the subject to the list of subject alternative names. So, make sure that you do it as part of your certificate request for your internal CA.

The reason you need to do this is because of how web browsers are processing subject alternative names. Web browsers are ignoring the subject if a list of subject alternative names is present. All web browsers seem to be enforcing this now. A few years ago, the web browsers would process both the subject and the subject alternative names.

Here's the quote from RFC 2818 from May 2000 (yes, that long ago):
If a subjectAltName extension of type dNSName is present, that MUST be used as the identity. Otherwise, the (most specific) Common Name field in the Subject field of the certificate MUST be used. Although the use of the Common Name is existing practice, it is deprecated and Certification Authorities are encouraged to use the dNSName instead.

As a slightly interesting side note. The subject field is optional when SANs are defined. However, some older software might still require the subject to be defined.

If you are configuring certificates in Exchange server, be aware that the Exchange admin console (EAC) does not properly differentiate between subject and subject alternative names. In the certificate shown below, webmail.adatum.com is in the subject and not in subject alternative names. EAC (Exchange 2016 CU8) is including the subject in the Subject Alternative Names box which can be misleading.


 

Wednesday, September 27, 2017

Automating Let's Encrypt DNS Verification with GoDaddy DNS for Exchange

The script that I reference in this post can be downloaded here: GoDaddyDNSUpdatePublic.ps1.txt

I love the concept of using Let's Encrypt for free SSL/TLS certificates. However, the short 90-day lifetime of the certificates is designed for automated renewal. In this blog post I'm going to show the steps required to script the use of GoDaddy for DNS verification.

For the basic steps on how to get a SAN certificate by using Let's Encrypt and DNS verification by using Windows PowerShell, please see my previous blog post: Using Let's Encrypt Certificates for Exchange Server

Let's Encrypt requires you to create an identifier for each DNS name that you want to include on a certificate. You need to validate each identifier to prove ownership of the domain. When you are using DNS validation, you need to create a TXT record in DNS for each identifier.

Unfortunately (from an ease of user perspective), the validation for an identifier is only valid for 30 days. This means, when you go to renew a certificate, you also need validate your identifiers again. Practically, this means you create new identifiers with the same DNS name, but a different alias, and validate them before generating the certificate.

Since DNS validation requires you to create a TXT record in DNS, you need a way to automate this. Fortunately, many DNS providers have a web API that you can use to programmatically access and create DNS records. However, be aware that there is no wide spread standard for this API. GoDaddy has created DomainConnect and submitted it as a standard, but I've not seen wide acceptance of it.

For this blog, I'll be showing the use of GoDaddy's API mostly because it's the DNS provider that I use most often.

Authentication to create and manage DNS is done by using an API key and a secret. Both of these are included in the URL when you perform tasks. You get the API key and secret from the GoDaddy Developer Portal (developer.godaddy.com).

On the GoDaddy Developer portal:
  1. Sign-in by using your GoDaddy Account
  2. Go to the Keys tab and click Create Key.
  3. Give your key a name.
  4. Copy the key and the secret to a file and click OK.
This creates a test key which you cannot use for updating your DNS. However, you're now at a screen where you can create a production key. Save the details of that production key to a file for using in the script.

At this point, it is assumed that you've already created a vault and registered by using the ACMESharp cmdlets. The remaining steps are purely to automate the process.

 #define domain that records are being created in  
 #script only supports a single domain  
 $domain = 'yourdomain.com'  
   
 #For Pfx file  
 $pfxPass = "PasswordOfYourChoiceToSecurePfxFile"  
 $pfxPath = "C:\Scripts\cert.pfx"  
   
 #header used for accessing GoDaddy API  
 $key = 'YourBigLongKeyHere'  
 $secret = 'YourBigLongPasswordHere'  
 $headers = @{}  
 $headers["Authorization"] = 'sso-key ' + $key + ':' + $secret  
   
 #First identity will be the subject, all others in SAN   
 $identities = "mail.yourdomain.com","autodiscover.yourdomain.com" 

 $allAlias = @()

I started my script by defining some variables used later on:
  • $domain is your domain in GoDaddy where the DNS records are being created.
  • $pfxPass and $pfxPath are used used then the certificate is exported to a PFX file before being imported into the Exchange Server.
  • $key and $secret are provided by GoDaddy when you obtain your production key for the API.
  • $headers is included as the authentication information later on when the call is made to the GoDaddy web api to create the TXT record.
  • $identities contains the DNS names that will be included in the certificate. My example has two names, but more names can be added.
  • $allAlias is defined as an array so that later functionality adding aliases can be processed.

 Foreach ($ident in $identities) {  
   [string]$unique = (Get-Date).Ticks  
   $alias = ($ident.Replace(".",""))+$unique  
   $allAlias = $allAlias + $alias  
   $id=New-ACMEIdentifier -Dns $ident -alias $alias   
   If ($id.Status -eq "valid") {Continue}  
   $chResults = Complete-ACMEChallenge $alias -ChallengeType dns-01 -Handler manual  
   $chData = $chResults.Challenges | Where-Object {$_.Type -eq "dns-01"}  
   $value=$chData.Challenge.RecordValue  
   $name=$chData.Challenge.RecordName  
     
   #remove domain name from $name  
   $recordname = $name.Replace(".$domain","")  

I use a Foreach look to create each identity and verify it using DNS. I'm going to go through this Foreach loop in chunks.

Since I'm going to need to create multiple identities over time, I wanted a unique identifier ($unique) to ensure there wouldn't be conflicts in naming. I chose to use the ticks value from time. This has the added advantage that you could sort them based on when they were created.

Each identity is referred by by it's alias. I defined the alias for each identity as the DNS name of the identity with the dots removed and $unique added. After each alias is generated, it's added to the $allAlias array.

When the identifier is created for Let's Encrypt, it's placed in the $id variable. The $id variable is then used to verify the status of the identifier. If an identifier with the same DNS name has already been created and verified then the status is valid. If it's valid, we don't need to do any of the other work in the loop and Continue to tell the script to carry on with the next identifier. If the status is not valid (which is expected for new identifiers) then we process the rest of the loop.

The results for the Complete-ACME challenge are placed in $chResults. The Challenges property for those results for the dns-01 challenge type are placed in $chData where we can get the RecordValue and RecordName properties:
  • $name contains the name of the TXT record required for validation
  • $value contains the text string that needs to be included in the TXT record for validation
When the TXT record is created by using the GoDaddy API, the data used in the request does not contain the domain name in the name. The $name variable is processed to remove the domain name (the domain name is replaced with nothing) and the results placed in $recordname which contains the data that will be submitted to the GoDaddy API.

 #create DNS record  
 $json = ConvertTo-Json @{data=$value}
 Invoke-WebRequest https://api.godaddy.com/v1/domains/$domain/records/TXT/$recordname -method put -headers $headers -Body $json -ContentType "application/json"  

UPDATE: July 2, 2018
A reader named Jason has reported that the json format used by GoDaddy has changed and that the above code snippet needs to be updated. I have not verified, but Jason says updating the middle line to the following fixes the problem:
  • $json = ConvertTo-Json @(@{data=$value})


After all the processing of data is done, creating the TXT record is fairly straightforward. The data for the request is put into a hash table that is converted into Json. This hash table only requires the data but you can include other information like the TTL.

Invoke-WebRequest accesses the GoDaddy web api with a put method to send the data. This same method that web forms use to return data to a web site. The URL being access needs to contain your domain name and the type of record being created. In this case, I hard coded TXT as the record type in the URL, but $domain is used to insert the domain name. The $recordname variable is included in the URL because we only want to create that specific record. If the URL ends at TXT then the API assumes that we're providing an array of all the TXT records and any other existing TXT records are wiped out. The $headers variable (defined earlier) provides the authentication information for the request.

 #Submit the challenge to verify the DNS record  
 #30 second wait is to ensure that DNS record is available via query  
 Do {    
   Write-Host "Waiting 30 seconds before testing DNS record"  
   Start-Sleep -Seconds 30
   $dnslookup = Resolve-DnsName -Type TXT -Name $name  
   $dnsSuccess = $false  
   If ($dnslookup.strings -eq $value) {  
     Write-Host "DNS query was successful"  
     Submit-ACMEChallenge $alias -ChallengeType dns-01  
     $dnsSuccess = $true  
   } Else {  
     Write-Host "DNS query for $name failed"  
     Write-Host "The value is not $value"  
     Write-Host "We will try again"  
   }  
 } Until ($dnsSuccess -eq $true)   
   

I ran into an issue when verifying the TXT records. After creating the TXT record there can be a delay (a few second to a few minutes) until the record is accessible via the DNS servers. If Let's Encrypt tries to verify before it accessible, then it fails and isn't recoverable. You need to create a new identifier. So, I added this code to verify the DNS record is accessible before telling Let's Encrypt to verify.

The Resolve-DnsName cmdlet looks for the TXT record we just created. If the lookup contains the expected value then Submit-ACMEChallenge is used to tell Let's Encrypt to verify it. Also, the variable $dnsSuccess is set to $true and the loop ends. If it's not successful, we try again at 30 second intervals until it resolves successfully. Since adding this code to the script I haven't had any failures from Let's Encrypt. I think there may be some caching of failed lookups on the client running the script which result in a two minute delay, but better that than the Let's Encrypt lookup failing.

 #Verify that the dns record was checked and valid  
 $chStatus = $null  
 Do {  
   Write-Host "Waiting 10 seconds before testing DNS validation"  
   Start-Sleep -Seconds 10    
   $chStatus = (Update-ACMEIdentifier $alias).Status   
   #$chStatus=((Update-ACMEIdentifier $alias -ChallengeType dns-01).Challenges | Where-Object {$_.Type -eq "dns-01"}).Status  
   If ($chStatus -eq "valid") {Write-Host "$ident is verfied"}  
   Else {Write-Host "DNS record not valid yet: $chStatus"}  
 } Until ($chStatus -eq "valid")  
   

After the DNS lookup for the TXT record is successful, the script uses Update-ACME to retrieve the status of the verification. There is a 10 second pause to allow Let's Encrypt to perform the verification. If the verification is still pending then the loop repeats again at 10 second intervals.

I have two methods for checking the status ($chStatus). The more complex version is one I saw in an example someone else provided. However, I found that the simpler version seems to work fine. However, I did see one person indicating that when the challenge type is not specified that a pending request is not properly retained in the local vault and fails. With my delay of 10 seconds, I'm not sure that's ever happened. Both versions do seem to work though.

This is the end of the Foreach loop that processes the identifiers. Each identifier has now been verfied and there is a variable $allAlias that contains the alias used for each identifier. Next up is creating the certificate.

 #Create Certificate  
 New-ACMECertificate $allAlias[0] -generate -AlternativeIdentifierRefs $allAlias -Alias $allAlias[0]  
 Submit-ACMECertificate $allAlias[0]  
 Write-Host "Waiting 10 seconds for certificate to be generated"  
 Start-Sleep -Seconds 10  
 Update-ACMECertificate $allAlias[0]  
 Get-ACMECertificate $allAlias[0] -ExportPkcs12 $pfxPath -CertificatePassword $pfxPass -Overwrite  
   

The New-ACMECertificate cmdlet creates a certificate request. The first identifier ($allAlias[0]) becomes the subject of the certificate. Then the entire array $allAlias is submitted as alternative references which is the Subject Alternative Names attribute in the certificate. When you list the alternative identifiers manually, you can skip the identifier used for the subject because it's automatically added to the SAN attribute also. However, it works fine when that identifier is specified also. The alias for the certificate is set to be the same as the alias for the identifier used as the subject.

The certificate request is submitted and the script waits for 10 seconds to ensure that Let's Encrypt has time to generate the certificate. Update-ACMECertificate retrieve the certificate information from Let's Encrypt and puts it in the local vault.

Get-ACMECertificate with the -ExportPkcs parameter is used to export the certificate to a PFX file that can be imported into Exchange Server. While a password is not required while overwriting, the import of a PFX file without a password won't work properly. All will appear good, but the certificate will behave as if it has no private key. The -Overwrite parameter is specified because it's assumed that this script will be automated and this allows the file generated to be overwritten each time.

 #Assign certificate using EMS commands  
 #If run as a scheduled task must be run at EMS prompt  
 $pfxSecurePass = ConvertTo-SecureString -String $pfxPass -AsPlainText -Force  
 $cert = Import-ExchangeCertificate -FileName $pfxPath -Password $pfxSecurePass -FriendlyName $allAlias[0] -PrivateKeyExportable $true  
 Enable-ExchangeCertificate $cert.Thumbprint -Services IIS,SMTP -Force  
   

Finally, we import the certificate into Exchange Server. The Import-ExchangeCertificate cmdlet won't accept a plain text password. So, the password is converted to a secure string that can be used and placed into $pfxSecurePass.

To make it easier to identify the certificate on the Exchange server, the alias is used as the friendly name. The private key is also marked as exportable because by default it is not.

After the certificate is imported, then Enable-ExchangeCertificate is used to specify that it should be applied to the IIS and SMTP services. The -Force parameter enables the cmdlet to complete without requiring user input. However, it will replace the default SMTP certificate with this option.

Because the Exchange cmdlets are used, you need to schedule this script so that it runs in the Exchange Management Shell and not a regular PowerShell prompt.

If you are going to use this certificate on multiple Exchange servers, then you should update this script to import and enable the certificate on all of the Exchange servers in the site. The current configuration assumes one Exchange server and it applies only on the local Exchange server where the script is run.

Is it worth it?

In reality, the complexity of keeping this up an running is probably not worth it for Exchange Server. Exchange Server is a mission critical part of your organization. Instead I'd go for a low cost certificate provider and get a cert that lasts for 3 years. A SAN certificate from NameCheap.com costs about $35US per year. That's far less than the value of your time getting this up and going.

That said, this was a fun exercise for me and I'll probably use this for test environments. Now that I have the script it's pretty easy for me to use it. 

Alternative for IIS

As part of developing this, I worked out a method to supply the certificate only to IIS. I don't think that it's very useful for Exchange Server since we also want it to apply to SMTP to allow secured SMTP communication. However, I'm including it in case anyone is interested.

 #Import certificate from pfx to server  
 #script must be running as administrator to perform this action  
 $cert=Import-PfxCertificate -FilePath $pfxPath -CertStoreLocation Cert:\LocalMachine\My -Password $pfxSecurePass -Exportable  
   
 #Assign certificate to default web site  
 Import-Module WebAdministration #required for IIS: PSDrive  
 Set-Location IIS:\SSLBindings  
 Get-ChildItem | Where-Object { $_.sites.Value -eq "Default Web Site" -and $_.Port -eq 443 -and $_.IPAddress.IPAddressToString -eq "0.0.0.0" } | Remove-Item  
 $cert | New-Item .\0.0.0.0:443  
   

Importing the script is done with the Import-PfxCertificate. Again, we specify a path, password, and mark it as exportable.

We manually import the WebAdministration module to get access to the IIS: PSDrive. In Windows Server 2012 and later, modules autoload for using cmdlets, but not PSDrives.

SSL bindings are located in IIS:\SSLBindings. So, the location is set to there. In that location, Get-ChildItem gets a list of the SSL bindings. The binding for all IP addresses on port 443 of Default Web Site is deleted.

The certificate information is piped to New-Item with .\0.0.0.0:443. This creates a new binding in the current directory for that certificate to 0.0.0.0 on port 443.

References:

Thursday, September 14, 2017

Getting Detailed Error Messages for Mailbox Moves

In Office 365 or Exchange Server 2013/2016, you can use the administration console to view information about migration batches. To find out information about failing moves, you can view the details of the migration batch and then view the report for individual mailboxes. When you view the report for a mailbox a text file is downloaded for viewing.

The report provides detailed information about how much data has been downloaded. Also, if there are errors, they are contained in the report. Unfortunately sometimes the errors are pretty generic. For example, one error I got recently was:
Transient error TimeoutErrorTransientException has occurred. The system will retry (200/1300).
Instructions on how to review the report:
Since the error was happening often, we needed to get more information. Fortunately that detail is available, but not in that report. Instead, you need to use Windows PowerShell to view the move request statistics. If you are moving the mailbox to Office 365, you need to use a PowerShell prompt connected to Exchange online.

$stats = Get-MoveRequest user@domain.com | Get-MoveRequestStatistics -IncludeReport
$moveErr = $stats.Report.Entries | Where-Object {$_.Type -eq "Error"}

This leaves you will an array named $moveErr that contains all of the errors. You can view each error by specifying an individual array item.

$moveErr[1]

In my case, I got this more detailed information in the Failure property:

TimeoutErrorTransientException: The call to 'https://hybridserver/EWS/mrsproxy.svc
hybridserver (15.1.1034.26 caps:07FD6FFFBF5FFFFFCB07FFFF)' timed out. Error
details: The request channel timed out while waiting for a reply after 00:00:30.9527011.
Increase the timeout value passed to the call to Request or increase the SendTimeout
value on the Binding. The time allotted to this operation may have been a portion of a
longer timeout. --> The request operation did not complete within the allotted timeout of
00:00:50. The time allotted to this operation may have been a portion of a longer
timeout. --> The request channel timed out while waiting for a reply after
00:00:30.9527011. Increase the timeout value passed to the call to Request or increase
the SendTimeout value on the Binding. The time allotted to this operation may have been a
portion of a longer timeout. --> The request operation did not complete within the
allotted timeout of 00:00:50. The time allotted to this operation may have been a portion
of a longer timeout.

That error information gave me enough information to start looking at timeout settings for the MRS proxy service.

Wednesday, June 28, 2017

Exchange 2010 OWA failure

Had a call about an older Exchange 2010 server this morning. Users were having problems working with Outlook Web App.

I saw a high volume of errors in that Application event log. This error was referring to both autodiscover and '/EWS/Exchange.asmx' and was appearing multiple times per minute.

Event ID: 3
Source: System.ServiceModel 3.0.0.0
Category: WebHost
WebHost failed to process a request.
 Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/32001227
 Exception: System.ServiceModel.ServiceActivationException: The service '/Autodiscover/autodiscover.xml' cannot be activated due to an exception during compilation.  The exception message is: This collection already contains an address with scheme http.  There can be at most one address per scheme in this collection.
Parameter name: item. ---> System.ArgumentException: This collection already contains an address with scheme http.  There can be at most one address per scheme in this collection.
There was also this error about OWA.

Event ID: 108
Source: MSExchange OWA
Category: Configuration
Outlook Web App couldn't connect Exchange Web Services due to a configuration error. Response code = "null, webException.Status = SendFailure".
The errors in the log began at about 12:15am when someone had done a server reboot. So, something in that reboot triggered the issue. My first thought was updates, but no updates had been applied recently.

After a bit of research, it turned out to be the bindings for the default web site causing the issue. At some point, someone had added additonal http and https bindings to the default web site. When I removed the additional http and https bindings everything started properly and the errors no longer appeared.

I am guessing that someone added those bindings a long time ago, but they didn't start causing the issue until IIS was restarted as part of the reboot.

Friday, June 23, 2017

Errors on Public Folder Migration

As I was doing a public folder migration today, I got a couple of errors that took me some time to resolve. These are caused by mail enabled public folders migrated from Exchange 2003. You will see these errors when you run Get-MailPublicFolder on Exchange 2010. Some of these errors will show up in the public folder migration logs when migrating to Exchange 2016. So, I prefer to clean these up first to ensure migration is successful.


Error #1

WARNING: The object domain.com/Microsoft Exchange System Objects/PF Name has been corrupted, and it's in an inconsistent state. The following validation errors happened:
WARNING: Could not convert property OnPremisesObjectGuid to type Guid. Byte array for GUID must be exactly 16 bytes long.
My best guess is that this property is left over from Exchange 2003 (or maybe earlier). The quick fix is to disable mail for the public folder and then mail-enable it again. However, when you do so, verify the email addresses before and after.

Error #2


WARNING: The object domain.com/Microsoft Exchange System Objects/PF Name has been corrupted, and it's in an inconsistent state. The following validation errors happened:
WARNING: Property expression "PF Name" isn't valid. Valid values are: Strings formed with characters from A to Z (uppercase or lowercase), digits from 0 to 9, !, #, $, %, &, ', *, +, -, /, =, ?, ^, _, `, {, |, } or ~. One or more periods may be embedded in an alias, but each period should be preceded and followed by at least one of the other characters. Unicode characters from U+00A1 to U+00FF are also valid in an alias, but they will be mapped to a best-fit US-ASCII string in the e-mail address, which is generated from such an alias.

This error is most commonly caused by a space in the Alias property. Update this property to remove spaces and the error should be gone.

Saturday, June 17, 2017

Multiple Moderation Approval Requests

I recently did a migration from Exchange 2010 to Exchange 2016 where the client uses a high volume of moderated messaging. There were over 100 transport rules that did message moderation of some sort. The initial deployment consisted of Exchange 2010 SP3 RU17 and Exchange 2016 CU4.

Deployment of Exchange 2016 into the Exchange 2010 environment didn't seem to have any effect. However, after we directed the internal namespace to Exchange 2016 for proxying, the approvals generated by the transport rules when whacky (yep that's the technical term).

Here is the process we saw:
  1. Message requiring moderation sent.
  2. Approval request sent to moderator.
  3. Moderator approves request
  4. Approval request sent to moderator
  5. Moderator approves request
  6. Repeat request and approval process a few more or a lot more times.
This process was happening even though we had not moved any mailboxes to Exchange 2016 yet. 

When searching, there were very few references to this issue on the Internet or support forums. However, there were a few suggestions that were consistent:
  • Ensure arbitration mailboxes are moved to Exchange 2016 (one of these stores the messages until they are approved).
  • Delete and recreate rules.
  • Move moderators mailbox to Exchange 2016.
  • Restart transport services.
All of these things were done but we still had issues. However, when both mailboxes were on Exchange 2016, the approvals on messages seemed to only happen twice. This was better than the random number from before.

I reviewed the message tracking logs for errors and didn't see any. In the logs, each time an approval was received, the message was released for delivery, but then promptly moderated again. However, the second attempt to approve worked.

All of my initial testing was done using inbound messages. So, I tried some scenarios with both mailboxes on Exchange 2016, and these were my results:
  • Inbound messages routing through Exchange 2010 first - Two approvals required
  • Outbound messages routing out through Exchange 2010 - One approval only
  • Inbound messages routing through Exchange 2016 only - One approval only
Based on these results, we can see that Exchange 2010 coexistence definitely plays a role, because when Exchange 2010 is not part of the inbound routing the issue doesn't occur. This at least provided confidence that after migration was complete the issue would not be persist.

The other item that needed to be addressed was the Exchange 2016 CU4. Microsoft releases updates in matched sets and Exchange 2016 CU4 was one step behind Exchange 2010 SP3 RU17. In the hope that having both at the same update level would fix it, we applied Exchange 2016 CU 5. We did the update to CU 5 but there was not change.

Final testing indicated that any message that entered the Exchange organization through Exchange 2010 was subject to double approval. This happened if the message came in from external or was generated by a mailbox in Exchange 2010. The location of the moderator mailbox did not make a difference.

So, to minimize the issue:
  • Move all inbound message routing to Exchange 2016 sooner rather than later. This includes Internet mail and applications that send messages to be moderated.
  • Move mailboxes that generate the most messages to be moderated first. Once the source is in Exchange 2016 the problem is mediated.
I should also note that there was a red herring in the application event log. We saw this error:
Event ID 1051, MSExchange Extensibility
Warning, MExRuntime
 Agent 'Approval Processing Agent' caused an unhandled exception 'SmtpResponseException: 250 2.1.5 APPROVAL.ApprovalRequestUpdated; approval request updated successfully' while handling event 'OnCreatedMessage'
However, review of the logs indicated that the error was present before the issue appeared. So it appears to be noise rather than useful information.

Friday, May 12, 2017

Everyone Dynamic Group with Office 365 Hybrid

Many organizations have configured a dynamic distribution group that includes all users with mailboxes. This Everyone group is used for sending out company notifications and the like. Because this group is dynamic, it's automatically updated as new mailboxes are added and removed.

When you implement hybrid mode with Office 365, the dynamic group will include on-premises mailboxes but not cloud mailboxes. This is because cloud mailboxes are a different recipient type that is not included. Cloud mailboxes are MailUser recipients (an AD user with an email address but no mailbox).

One way to fix this is to modify the dynamic distribution group to include MailUser recipients. The screenshot below has enabled Users with external e-mail addresses.


This will include users with cloud mailboxes. This is the recipient filter for the dynamic distribution group:
((((RecipientType -eq 'UserMailbox') -or (RecipientType -eq 'MailUser'))) -and (-not(Name -like 'SystemMailbox{*')) -and (-not(Name -like 'CAS_{*')) -and (-not(RecipientTypeDetailsValue -eq 'MailboxPlan')) -and (-not(RecipientTypeDetailsValue -eq 'DiscoveryMailbox')) -and (-not(RecipientTypeDetailsValue -eq 'ArbitrationMailbox')))
Note that everything after the two recipient types is added automatically when you set the filter. So, don't include that information when you set a filter. If you do, those requirements will be in there twice.

Unfortunately, the recipient type UserMailbox includes on-premises resource mailboxes and the MailUser recipient type includes cloud resource mailboxes. To exclude those from our dynamic distribution group we need to manually configure the filter. We can do this by excluding the following values for RecipientTypeDetails:
  • RoomMailbox
  • EquipmentMailbox
  • RemoteRoomMailbox
  • RemoteEquipmentMailbox
I set the following filter to achieve this:
((((RecipientType -eq 'UserMailbox') -or (RecipientType -eq 'MailUser'))) -and (-not(RecipientTypeDetailsValue -eq 'RemoteRoomMailbox')) -and (-not(RecipientTypeDetailsValue -eq 'RoomMailbox')) -and (-not(RecipientTypeDetailsValue -eq 'RemoteEquipmentMailbox')) -and (-not(RecipientTypeDetailsValue -eq 'EquipmentMailbox')))
Now, the next challenge is allowing cloud users to see this dynamic distribution group. This is an issue because Azure AD Connect does not synchronize dynamic distribution groups to Office 365. For more information about this see:



Wednesday, May 10, 2017

Removing a Cloud Mailbox from A Synced User

The title for this post is a bit incomplete because there wasn't enough space for a full explanation. Here is the scenario I had...

Before setting up hybrid mode for an Exchange 2010 organization, a user account was created in Office 365 with the same UPN as a user account in on-premises. The on-premises domain had been added to Office 365, but directory synchronization was not yet in place.  As part of testing, the user account had been give an Office 365 license, which then created an online Exchange mailbox.

When directory synchronization was configured, the on-premises AD account was matched with the existing cloud user. However, the on-premises user has an on-premises mailbox and the cloud user has a cloud mailbox. When viewing the cloud user in Office 365, you could see the mailbox in the cloud.

Because there was an existing cloud mailbox I couldn't move the on-premises mailbox up to Office 365. There was also no way to remove the cloud mailbox to allow the on-premises mailbox to replace it. If the user license was removed, then the cloud mailbox was removed, but came back again when the license was re-added.

To fix this I had to delete the cloud user account and recreate it. To delete the cloud user account, I moved it into an organizational unit that Azure AD Connect was not syncing. Then I purged the user account in Office 365 with the following cmdlet:
Remove-MsolUser -UserPrincipalName user@domain.com -RemoveFromRecycleBin
Finally I moved the user back to an OU that was synchronized. This recreated the cloud user account without a mailbox and the account was identified as having an on-premises mailbox.

Monday, May 8, 2017

New Hybrid Free/Busy Fails Cloud to On-Prem

When you configure hybrid mode, calendar sharing is automatically configured between on-premises and cloud users. You don't need to configure anything. However, in some cases, you may need to perform an IISreset.

On a recent project implementing implementing hybrid mode for an Exchange 2010 organization, we enabled hybrid mode and tested connectivity. Everything looked good for mail flow and mailbox moves. However, free/busy lookups failed from the cloud users to on-premises. When we tried to view the on-premises users' calendars the status was "No Information".

When you're not sure what's going on, it's always a good idea to use the Remote Connectivity Analyzer to test things out. There is an Office 365 tab which includes a Free/Busy test. When I ran this test, it was successful, as shown in the figure below. However it still wasn't working.

It's been a while since I've implemented Hybrid for Exchange 2010. With a quick search, I found this blog post that reminded me that an IIS reset will fix this up:
Most Exchange IIS related issues resolve themselves after a period of time. That period of time might be 12-16 hours, but usually there is some background process that refreshes things.

In this case, hybrid mode had been configured for multiple days. The free/busy lookups required an IISreset or a server reboot to fix.

This fix is certainly relevant for Exchange Server 2010. It may also be relevant for later versions of Exchange Server.

Wednesday, April 26, 2017

Must Run O365 Hybrid Wizard with IE

Today I made the mistake of downloading and attempting to run the Office 365 Hybrid Deployment Wizard by using Firefox. When you do, the app starts to launch and the fails. When you click on the Details button to open the log file, you see this under the error summary:
Deployment and application do not have matching security zones.
Download the app from within IE and no problems.

Link to download the wizard:
UPDATE: Also worth noting that a similar error with the wizard not running occurs if the .application file extension is not associated with Internet Explorer.

Tuesday, April 18, 2017

Script to Synchronize Primary Email Address with UPN

When planning an Office 365 implementation, it is best practice to start by assuming that UPN for signing in to Office 365 should match the user email address. If you don't configure it this way, then users have two separate items (their UPN for signing in and their email address) that look very similar. In many cases users are confused by the similarity.

If you are synchronizing  your on-premises Active Directory with Office 365 (in most cases you do) then you need to set the UPN for the on-premises user accounts with the correct UPN. The UPN from on-premises user accounts is synchronized to Office 365 to create the ID for signing in.

Most organizations are not using the UPN on user accounts for authentication on-premises. The option has been there since Windows 2000, but most organizations still use the domainname\username format for authentication. However, you need to verify if any user accounts are using the UPN for authentication before making this change. At minimum, you should communicate with your application and system administrators to see if they are aware of anything that might use UPNs. If your organization has issued certificates to users, they might be using UPN as the unique identifier for the certificate.

The script below does the following:
  • Obtains a list of all users where the proxyAddresses attribute has a value. This is done so that the result include only user accounts with an Exchange attributes configured.
  • Identifies the primary email address based on the all caps "SMTP:" text.
  • Strips out the "SMTP:" text from the primary SMTP address.
  • If the new UPN and the existing UPN do not match the user account is updated and the change is logged.
The location and name of the log file are configured in $logfile. You need to manually configure this variable and verify that the necessary folders exist.

  
 #Log folder must already exist  
 $logfile = "C:\Scripts\SyncUPN.txt"  
   
 #Adds timestamp to log file  
 Get-Date | Out-File -FilePath $logfile -Append  
   
 #Obtains only users with valid proxyAddresses attribute   
 $users = Get-ADUser -Properties proxyAddresses -Filter {proxyAddresses -like "*"}  
   
 #Prepare variables for processing status  
 $total = $users.count  
 $current = 0  
   
 Foreach ($u in $users) {  
   #Find primary SMTP address for user  
   $primarySMTP = $u.proxyAddresses | Where-Object {$_ -clike "SMTP:*"}  
       
   #Remove "SMTP:" to create the new UPN value  
   $newUPN = $primarySMTP.Substring(5)  
       
   #Set the new UPN value only if required  
   If ($u.UserPrincipalName -ne $newUPN) {  
     $u.DistinguishedName + " Old UPN: " + $u.UserPrincipalName | Out-File -FilePath $logfile -Append  
     $u.DistinguishedName + " New UPN: " + $newUPN | Out-File -FilePath $logfile -Append  
     Set-ADUser $u -UserPrincipalName $newUPN      
   } #end if  
   
   #Processing status  
   $current += 1  
   Write-Progress -Activity "Processing users to update UPN to primary email address" -Status "Progress: $current" -PercentComplete ($current/$total*100)   
        
 } #end foreach  

Script to Remove Old Domains from User Email Addresses

When managing email addresses and domains in Exchange Server, old email addresses are never removed automatically. This is good because it ensures that email addresses on a mailbox are never accidentally lost. However, you may want to clean up old domains or address formats that are no longer in use.

Some common scenarios where you might want to remove an old domain:
  • An SMB deployment of Exchange Server where a .local domain was added as the first domain for email addresses.
  • Old GroupWise addresses are left in place from an older migration.
  • Obsolete domain left over from a company merger many years ago
I often find that obsolete domains are identified when I run IDFix as part of preparing to migrate to Office 365. To simplify the removal of obsolete domains, I have created the following script.

A few things to note:
  • You need to set $RemovePattern to identify the domain to be removed. Any email addresses matching this pattern will be removed from proxyAddresses attribute in Active Directory objects.
  • The script uses Get-ADObject rather than Get-ADUser to make sure that the domain is removed from distribution groups too.
  • This version of the script is capable of removing multiple instances of a matching email address. So, if a user has several email addresses in the old domain, all of them are removed.
  • At the end of the script, I use Write-Progress to display a status bar. It's not necessary, but if there is a large number of users it's nice to see activity on the screen instead of just waiting and hoping it's doing something.

 #This pattern is used to match the email addresses being removed.  
 #Test that this pattern finds the correct users and email addresses  
 #before running this script.  
 #Example: $pattern = "smtp:*@olddomain.com"  
 $RemovePattern = "smtp:*@olddomain.local"  
   
 Import-Module ActiveDirectory #only required for 2008 R2  
   
 #Get the users that have an email address that matches the pattern  
 Clear-Host  
 Write-Host "Querying objects...This may take a moment"  
 Write-Host ""  
 $objects = Get-ADObject -Filter {ProxyAddresses -like $RemovePattern} -Properties ProxyAddresses  
   
 #Identify address being removed from first user for warning  
 [String]$proxyexample = $objects[0].proxyaddresses -like $RemovePattern  
   
 #Display warning and get confirmation  
 Write-Host "You are going to remove email addresses that match the following pattern:"  
 Write-Host -ForegroundColor Red "$RemovePattern"  
 Write-Host ""  
 Write-Host "This is an example from the first object:"  
 Write-Host -ForegroundColor Red "$proxyexample"  
 Write-Host ""  
 Write-Host "This will modify $($objects.count) objects"  
 Write-Host ""  
   
 $confirm = Read-Host "Enter Y to continue"  
 If ($confirm -ne "Y") {Break}  
   
 #Prepare variables for processing status  
 $total = $objects.count  
 $current = 0  
   
 #Processing users to remove addresses  
 Foreach ($o in $objects) {  
     
   #Build list of addresses to remove for object  
   #required because there might be multiple that match  
   $proxy= New-Object System.Collections.ArrayList  
   
   Foreach ($a in ($o.ProxyAddresses)) {  
     If ($a -like $RemovePattern) {  
       $proxy.add($a) | Out-Null  
     } #end if  
   } #end foreach  
     
   #Remove each bad address  
   Foreach ($p in $proxy) {  
     Set-ADObject $o -Remove @{'proxyAddresses'=$p}  
   }  
     
   #Pause  
   
   #Processing status  
   $current += 1  
   Write-Progress -Activity "Removing email addresses that match pattern" -Status "Progress: $current" -PercentComplete ($current/$total*100)   
 } #end foreach  

Thursday, March 9, 2017

Exchange 2010 SP3 Hub Transport Upgrade Error

Ran into a new issue yesterday related to installing Exchange 2010 SP3. I was called in to help when the initial upgrade attempt failed. The error during SP3 installation was:
An unexpected error occured while modifying the forms authentication settings for path /LM/W3SVC/1. The error returned was 5506.
A screenshot of the error is below:


Doing a search didn't come up with much, but it did give this:
That link seemed to indicate that it could be related to the SSL binding on the default web site in IIS. Taking a look at the SSL binding, it seemed to be missing the certificate assignment. However, when I tried to add the certificate I got a strange error about the session be closed.

Ok then, since you won't let me add the SSL certificate to the binding by using IIS Manager, let's try with Exchange Admin Console. When I assigned the IIS service to the certificate in EAC, it all looked fine. I also took this moment to review the certificate and verify that the SAN names were correct. I also noted that it did indicate that there was a private key for the certificate.

After this the binding worked because we could access https://servername/owa URL, but it returned a 503 error. However, rather than attempting to fix that error, we tried the SP3 install again. Since a service pack upgrade rewrites a lot of the content in the IIS virtual directories we through we might get lucky and it would fix and configuration errors that we had.

During the next install, the installation of the Hub Transport role completed successfully, but now we got an error on the Client Access role installation. As we were actively troubleshooting I didn't write it down at the time, but it was something like:
Could not grant Network Service access to the certificate with thumbprint BIGLONGHEXTHUMBPRINT because...
Based on this I decided to review the certificate in the Certificates MMC snap-in. Again, all the details looked right. Maybe I can add the necessary permissions myself for Network Service. To access the permissions for a certificate, you right-click it, point to All Tasks, and click Manage Private Keys. This normally brings up a security dialog box. However for me it brought up the following error:
Object not found.
I interpreted this error to mean that either the private keys were not really present for the certificate. Or the Domain Admin account that we were using to access the certificate and run the install didn't have permission to access the private keys. In either case, since our Domain Admin account couldn't set permissions in the certificate, we were dead in the water.

Fortunately certificates are much less expensive than they used to be and we quickly obtained a new certificate with all of the necessary names from NameCheap. They might not have the best management tools for certificates, but the price is right. So, if this didn't work it didn't waste a lot of money.

After installing the new certificate and assigning the correct services to it, we ran the Exchange 2010 SP3 upgrade again. And after some nervous waiting, the upgrade completed properly. And the upgrade fixed all of the errors for the web services. Email for phones began to work immediately, as did OWA.

Sunday, January 15, 2017

Exchange VM Hangs During Updates

I haven't run into this yet, but it appears that in some cases, Hyper-V virtual machines running Exchange Server will hang when installing updates. Specifically this seems to occur when running updates for Hyper-V integration services.

There are reports of KB3037623 specifically causing this issue.
The fix is to:
  1. Disable the Exchange services
  2. Apply the update
  3. Reenable the Exchange services
This blog posting provides detailed steps:

Thursday, October 6, 2016

Exchange Server Unable to Verify CRL

Just recently ran into another issue related to certificate revocation list (CRL) verification. This time it was an Exchange 2010 organization that had been fine when we initially installed the certificates, but now in the Exchange Management Console was showing the currently assigned certificate with a red X and an error message indicating that the CRL for the certificate could not be verified.

The certificate was still valid, but the Exchange server couldn't verify that it hadn't been revoked. No clients were affected by this issue. Viewing the certificate on a client accessing OWA showed as valid.

Like many organizations, this organization has a proxy between the internal network and the Internet. For the Exchange Servers to verify the CRL, they need to download it from the source specified in the certificate. This had been working, so, what changed?

It turns out that as part of troubleshooting connectivity to WSUS from the Exchange servers, the proxy configuration was removed. The connectivity for CRL verification is handled by the operating system. So, we needed to setup the proxy again.

The simplest way to configure the proxy for the operating system is to first configure the proxy settings in IE and them import them. After configuring IE proxy settings properly, use the following steps:
  1. Open a command prompt.
  2. Type netsh and press Enter.
  3. At the netsh prompt, type winhttp and press Enter.
  4. Type import proxy source = ie and press Enter.
For some other notes about Exchange Server and proxies, see: http://byronwright.blogspot.ca/2014/09/firewalls-and-proxy-for-exchange-hybrid.html

For some additional info about verifying certificates, see: http://byronwright.blogspot.ca/2016/10/expired-offline-root-ca-crl-causes.html

Sunday, July 3, 2016

Manual Reseed of Exchange Content Index

I recently ran into an issue where a mailbox database content index was in a failed state after the Exchange 2010 server had a power outage. Not too big of a deal since it was in a remote site that was hosting only passive databases for disaster recovery.

Normally for a passive copy, you run the following command and all is good:
Update-MailboxDatabaseCopy -Identity DBName\ServerName -CatalogOnly

However I got this error:
A source-side operation failed. Error An error occurred while performing the seed operation. Error: An error occurred while updating the search catalog files from server 'XXXXXXX'. Error: A transient exception from Exchange Search was encountered. Error: Catalog was not paused for indexing on database dbGUID.

There were only a couple of search hits for this specific error about pausing indexing and they really only indicated it was an internal process as part of the catalog reseed. There were no useful errors in the event log.

I tried a few different variations with no success:
  • initiate from passive
  • initiate from active
  • suspend passive copy before updating catalog
  • remove catalog files on passive before starting
  • force AD replication in case replication lag was causing an issue
Finally, what worked was the following:
  1. Stopped the Microsoft Exchange Search Service on both the active and passive copies.
  2. Deleted the catalog folder on the passive.
  3. Manually copied the catalog folder from the active to the passive.
  4. Started the Microsoft Exchange Search Service on the active.
  5. Started the Microsoft Exchange Search Service on the passive.
After following this process, the content index on the passive copy is good.