Sunday, October 9, 2016

Modifying Mailbox Replication Latency Tolerance for Mailbox Moves

During the summer I was doing an Exchange 2013 to Exchange 2016 migration. All was going well except that mailbox moves would stall out and take a very long time. I didn't capture the logs at the time, but when looking at the logs, it indicated that the mailbox moves were stalling due to disk health and more specifically, disk latency.

To view the status of individual mailbox moves that are not completed, use the following command:

 Get-MoveRequest | Where {$_.Status -ne "Completed"} | Get-MoveRequestStatistics  

The default output for Get-MoveRequestStatistics only shows the DisplayName, StatusDetail,  TotalMailboxSize, TotalArchiveSize, and PercentComplete. However, there are some other useful properties in there for stalled moves such as:
  • TotalStalledDueToContentIndexingDuration
  • TotalStalledDueToMdbReplicationDuration
  • TotalStalledDueToMailboxLockedDuration
  • TotalStalledDueToReadThrottle
  • TotalStalledDueToWriteThrottle
  • TotalStalledDueToReadCpu
  • TotalStalledDueToWriteCpu
  • TotalStalledDueToReadUnknown
  • TotalStalledDueToWriteUnknown
  • DiagnosticInfo
  • Report
In my case a slow mailbox move was showing stalling due to Content Indexing Duration (caused by slow disk) and Write Throttle (triggered by disk latency).

To make things easier I tend to dump everything into a variable so that I can look at them different ways and one by one. The example below collects all of the move request statistics in a single variable and then displays the all of the properties for the first move.

 $moves = Get-MoveRequest | Where {$_.Status -ne "Completed"} | Get-MoveRequestStatistics  
 $move[0] | Format-List   

Now the preferred option is to figure out why your disk latency is high and try to resolve that. However, in this scenario, we needed to get the mailboxes moved because there were too many other things on the SAN to tinker with.

So, to speed up the mailbox move process, I tweaked the configuration file for the Mailbox Replication service to allow a higher level of latency before stalling out on the moves. This file is C:\Program Files\Microsoft\Exchange\v15\bin\MsExchangeMailboxReplication.exe.config.

At the end of this file there are a number of settings that control the service. For disk latency the settings are at the very end in the appSettings section. As shown below:

   <!-- Mdb latency health threshold values in msec. Valid range is from 0-1000. -->  
   <add key="MdbFairUnhealthyLatencyThreshold" value="20"/>  
   <add key="MdbHealthyFairLatencyThreshold" value="10"/>  
   <!-- Maximum delay that WLM returns under "Fair" database resource health in msec. Valid range is from 0-60000 -->  
   <add key="MdbLatencyMaxDelay" value="60000"/>  
   <add key="LogEnabled" value="true" />  
   <add key="LogDirectoryPath" value="%ExchangeInstallDir%Logging\MailboxReplicationService\" />  
   <add key="LogFileAgeInDays" value="30" />  
   <add key="LogDirectorySizeLimit" value="100MB" />  
   <add key="LogFileSizeLimit" value="10MB" />  
   <add key="LogCacheSizeLimit" value="2MB" />  
   <add key="LogFlushIntervalInSeconds" value="60" />  

The two settings for disk latency are MdbFairUnhealthyLatencyThreshold and MdbHealthyFairLatencyThreshold which have default values of 10ms and 20ms. In most cases those are reasonable defaults. I incrementally increased the values until the mailbox moves stopped stalling. My case was extreme and I ended up using values of 150ms and 75ms. The lower value of 75ms latency seemed to trigger the stalling.

When the config file is modified, you need to restart the Microsoft Exchange Mailbox Replication service to take effect. This will stop all mailbox moves that are in progress. They will start up again after the restart is complete but you may lose the progress that has already been made. For example, if 1.5 GB of a 2 GB mailbox has been moved, the whole mailbox move might start over again.

A couple more articles that could be helpful if you're looking at modifying this config file:

Thursday, October 6, 2016

KB3161949 Breaks Scan to Folder (and more...)

For small businesses, KB3161949 should not be a cause for concern. But for larger businesses with multiple subnets (don't need to be too large for that), KB3161949 might break multiple services that require NetBIOS over IP.

Your first thought is probably: Who the heck still uses NetBIOS over IP? I know that Windows doesn't need it anymore.

And, you're right. Windows (at least Vista and later) work just fine without NetBIOS over IP. However, there are a lot of devices with older SMB implementations that do require NetBIOS over IP. I ran into one Sharp MFP device today that requires it for scan to folder functionality.

Information about the update is published at:

This update is a security update for WPAD. WPAD is a method for automatically configuring devices with a proxy server. However, as part of locking down this process, it does one critical thing. Here's the quote:
NETBIOS communication outside of the local subnet is hardened. Therefore, by default, some features that depend on NETBIOS (such as SMB over NETBIOS) will not work outside the local subnet.
So, communication on the local subnet works, but if the MFP and server hosting the share are on different subnets then it breaks.

Fortunate the fix is pretty quick, add a registry key and reboot the server hosting the share:
Value Name: AllowNBToInternet
Type: Dword
Value: 1
Another Microsoft page that describes the same update, but slightly different description:

It's also worth noting that when troubleshooting this, the Security event log did not contain failed logon attempts for the user accounts. NetBIOS over IP was being dropped at the networking level and authentication was never attempted. 

Exchange Server Unable to Verify CRL

Just recently ran into another issue related to certificate revocation list (CRL) verification. This time it was an Exchange 2010 organization that had been fine when we initially installed the certificates, but now in the Exchange Management Console was showing the currently assigned certificate with a red X and an error message indicating that the CRL for the certificate could not be verified.

The certificate was still valid, but the Exchange server couldn't verify that it hadn't been revoked. No clients were affected by this issue. Viewing the certificate on a client accessing OWA showed as valid.

Like many organizations, this organization has a proxy between the internal network and the Internet. For the Exchange Servers to verify the CRL, they need to download it from the source specified in the certificate. This had been working, so, what changed?

It turns out that as part of troubleshooting connectivity to WSUS from the Exchange servers, the proxy configuration was removed. The connectivity for CRL verification is handled by the operating system. So, we needed to setup the proxy again.

The simplest way to configure the proxy for the operating system is to first configure the proxy settings in IE and them import them. After configuring IE proxy settings properly, use the following steps:
  1. Open a command prompt.
  2. Type netsh and press Enter.
  3. At the netsh prompt, type winhttp and press Enter.
  4. Type import proxy source = ie and press Enter.
For some other notes about Exchange Server and proxies, see:

For some additional info about verifying certificates, see:

Tuesday, October 4, 2016

Expired Offline Root CA CRL Causes Warnings

Today at a client site I noticed an error in the System log of all the domain controllers:
Source: Kerberos-Key-Distribution-Center
Event ID: 29
Level: Warning
The Key Distribution Center (KDC) cannot find a suitable certificate to use for smart card logons....

This client isn't using smart cards, but there has been some certificate strangeness is the past. So, it's worth investigating.

The most common reason for this error is a missing or invalid domain controller certificate. If the organization has an internal CA then it should automatically get a domain controller certificate from that CA. If the organization does not have an internal CA, then you can consider the warning cosmetic and ignore it.

In this case, the organization does have an internal CA. In fact, they have an offline root CA and an enterprise issuing CA.

Each domain controller has a certificate that is issued from an LDAPoverSSL certificate template. This certificate ensures that all of the domain controllers are properly configured to respond to LDAPS queries from applications.

When I look at the LDAPoverSSL certificate on a couple of servers, they are not expired. And when I view them in the Certificates MMC snap-in they are identified as valid. So, all looks good

I also verified that these certificate did have the necessary options enabled for smart card authentication. In the Enhanced Key Usage field, the following were listed:
  • KDC Authentication
  • Smart Card Logon
  • Server Authentication
  • Client Authentication
The Event ID text mentions verifying the certificate by using certutil.exe. So, I find a couple of web sites that recommend running certutil -verify, but this requires you to have the certificate in a file. Instead, you can run the following command on the server containing the certificate you want to check:
certutil.exe -verifystore MY > cert.txt
This command verifies all of the certificates in the local certificate store of the computer. Note that you can also verify user certificates by adding the -user option, but computer certificates are check without any option. The output from the command is quite long and I'm dumping it to a text file for easier reading.

The output contains information for each certificate in the store. In my case, I identified the LDAPoverSSL certificate in the output and found the following at the end of the data:
The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613)
Revocation check skipped -- server offline
Certificate is valid
Finally, a clue. So, I opened up the certificate and looked at the CRL Distribution Points field. This provided both an HTTP URL and an LDAP URL. When I accessed the HTTP URL, I was able to download and view the CRL without issue.

Certificate trust requires that the entire certification path is valid. So, next I looked at the issuing CA certificate in the chain. This certificate was also valid but pointed at a different CRL (same server, but different CRL file name for the offline root CA). When I downloaded this file I saw that Next Update field was two days ago. This corresponds within 30 minutes of when the warning began appearing in the event log.

To verify the validity of the certificate, the CRL for all certification authorities in the certification chain must be valid. Because the CRL from the offline root CA was expired, the certificate was not being trusted for all purposes. The fix is to update the CRL from the offline root.

We completed the following process to update the CRL:
  1. Power on the offline root CA
  2. Verify that the offline root CA updated the CRL at startup
  3. Copied the new CRL from the offline root CA to the CRL distribution point
  4. Published the new CRL to AD for availability via LDAP
To publish the CRL to Active Directory use the following command:
certutil -f -dspublish "CRL Path"
After this process was complete I verified the certificate again and did not get an error about skipping the revocation check.

Two important lessons:
  1. The Certificates MMC snap-in does not check the CRL for a certificate. So, to check certificate validity, use certutil.exe.
  2. If you have an offline root CA, you need to have a process in place to update the CRL from it.
Links with more information:

Tuesday, September 27, 2016

EOP not Moving Messages to Junk Email On-Premises

Exchange Online Protection (EOP) is Microsoft's solution for anti-spam and anti-malware. It is included as part of Office 365/Exchange Online and you can subscribe to it for on-premises Exchange.

When you implement EOP, you configure the MX records for your domain to deliver messages to EOP, and then EOP forwards to your Exchange server. If a message contains mailware or is obvious spam, it is typically blocked and not forwarded to your Exchange server. It can be quarantined in EOP or discarded.

Where is gets a bit tricky is the messages that might or might not be spam. They're spammy, but might be legitimate email. In Office 365, those messages are automatically moved to your Junk Email folder. For spammy messages to be moved to your Junk Email folder in on-premises Exchange, you need to create a couple of transport rules.

EOP adds an X-Forefront-Antispam-Report header to messages after they are evaluated. You need to create transport rules in your on-premises Exchange to read the value in this header and set the SCL (spam confidence level) value for the message. Exchange Server uses the SCL value to determine whether a message is moved to the Junk Email folder.

Microsoft indicates that you should create the following two rules:
New-TransportRule "EOPSpam1" -HeaderContainsMessageHeader "X-Forefront-Antispam-Report" -HeaderContainsWords "SFV:SPM"
-SetSCL 6
New-TransportRule "EOPSpam2" -HeaderContainsMessageHeader "X-Forefront-Antispam-Report" -HeaderContainsWords "SFV:SKS"
-SetSCL 6

Notice that these rules set the SCL value  to 6. Which means that, by default, Exchange Server will mark the messsages as spam and send them to the Junk Email folder.

However, I recently had a client where after configuring these rules, there were obvious spam messages still not going into Junk Email. At some point, this organization had changed the threshold for the SCL value that identifies spam. They had a value of 8.

So, when you implement these rules, you should also verify the SCLJunkThreshold configured for the Exchange organization. You can view the SCLJunkThreshold with the following command:
Get-OrganizationConfig | FL SCL*

If you need to change the SCLJunkThreshold, use the following command:
Set-OrganizationConfig -SCLJunkThreshold 6

Microsoft article about the creating the transport rules for Junk Email processing:
X-ForeFront-Antispam-Report Values:

Monday, September 26, 2016

Updated MCSE Certification for Exchange

I got a surprise email today indicating that I have a new Microsoft certification, the MCSE (Microsoft Certified Systems Expert): Productivity. This is the new MCSE certification that encompasses certification for Office 365, Exchange, Skype for Business, and SharePoint. The existing MCSE: Messaging is being retired March 31, 2017.

The overall MCSE certifications have been reorganized around the technical competencies that Microsoft partners are organized around. So, there are going to be less MCSE certifications with a wider focus:
  • MCSE: Productivity (includes Exchange)
  • MCSE: Cloud Platform and Infrastructure (Windows Server and Azure)
  • MCSE: Mobility (Windows 10 and Intune)
  • MCSE: Data Management and Analytics (SQL Server)
One of the big changes is how the MCSE ongoing certification is managed. The current MCSE: Messaging required recertification every 3 years to retain the MCSE. There was a specific recertification exam or you had the option to complete a series of MVA courses.

In the new MCSE: Productivity, you do not need to re-certify every three years, but you have the option to update your MCSE each year by taking a new elective exam. The desire for the new MCSE is that you are constantly improving you skills by adding to them each year. Effectively, now instead of getting a permanent MCSE, you get an MCSE for the year. And you can continue to get it yearly.

You also get more options for how to maintain your MCSE: Productivity. The base certification for the MCSE: Productivity is the MCSA: Office 365. Then you add a new elective each year. Some of the options are:
  • Designing and Deploying Microsoft Exchange Server 2016
  • Core Solutions of Microsoft Skype for Business 2015
  • Deploying Enterprise Voice with Skype for Business 2015
  • Core Solutions of Microsoft Exchange Server 2013
  • Advanced Solutions of Microsoft Exchange Server 2013
  • Managing Microsoft SharePoint Server 2016
  • Core Solutions of Microsoft SharePoint Server 2013
  • Advanced Solutions of Microsoft SharePoint Server 2013

Microsoft provides a FAQ about the new MCSE certifications here:

Details about the MCSE: Productivity are here:

And a blog posting on Born to Learn explaining the change:

Wednesday, August 31, 2016

Script to Create Migration Batches

Migration batches are a nice feature introduced in Exchange 2013 for managing mailbox moves. In general they work pretty well, but it can be a bit awkward to work with batches in large organizations. The graphical interface only displays 500 mailboxes at a time and this can be limiting.

To get the batches exactly as you want them, you often end up exporting a list of mailboxes to a CSV file and then cutting that down into batches. You can create a batch by importing a CSV file.

To simplify this process, I've created a script that takes all the mailboxes from a list of source databases and generates the CSV files in batch sizes that you specify. For example, you can use both Admin1DB and Admin2DB as your source databases. If you specify a batch size of 200 mailboxes, then you'll get CSV files with 200 mailboxes except for the final CSV which has the left overs.

The script can also automatically create the migration batches if you turn that option on. In the script, you can specify one or more destination mailbox databases for the moves.

The script does not specify an archive database. If your users have archive mailboxes, then you'll need to edit the migration batches after creation and specify the correct databases for the archive mailboxes.

A final note: The script creates the migration batches, but does not start them. Starting the migration batches and the timing of it is up to you.

The script is below. I hope you find it useful.

 #This script selects users from existing source mailbox databases  
 #and creates migration batches to new Mailbox databases  
 #Typical use is migration projects, but could also be used for  
 #server retirement or cleaning up corrupted mailbox databases  
 #Multiple migration batches are created but you need to start them.  
 #Archive mailboxes are not included when the migration batches are  
 #created. To move archive mailboxes, edit the migration matches  
 #after they are created.  
 #Created by Byron Wright (@ByronWrightGeek)  
 #Set variables for creating batches  
 #BatchSize is the number of mailboxes in each batch  
 #BatchName is text added on to the CSV name and batch name  
 #This is useful to uniquely identify groups such as  
 #Admin or Students  
 #Path for CSV files  
 #This patch must already exist  
 #Mailboxes in the the source mailbox databases that are put into migration batches  
 #One or more databases can be used  
 $SourceMbxDB="Mailbox Database 0999986598","VIP Users"  
 #When $CreateMigrationBatch is $true a migration batch is created from  
 #each CSV file. If it is $false then only the CSV files are created.  
 #If $false then you don't need to configure the destination database  
 #Batches move mailboxes to the destination mailbox databases  
 #One or more databases can be specified  
 #When multiple databases are specified, the mailboxes are  
 #spread among the databases based on number of mailboxes and  
 #not size of mailboxes.  
 $DestinationDB="DB1","Mailbox Database 1840440945"  
 #Get list of mailboxes to move  
 Foreach ($s in $SourceMbxDB) {  
      $Mbx+=Get-Mailbox -Database $s  
 #Add a batch number property to each mailbox  
 #Add the EmailAddress property to each mailbox as required for the New-MigrationBatch CSV  
 Foreach ($m in $Mbx) {  
     $m | Add-Member -NotePropertyName Batch -NotePropertyValue $batch  
     $m | Add-Member -MemberType AliasProperty -Name EmailAddress -Value PrimarySMTPAddress  
 #CreateCSV files and migration batches  
 For($b=1;$b -le $TotalBatches;$b++) {  
      $mbx | Where-Object Batch -eq $b | Select-Object Name,EmailAddress,Batch | Export-CSV $BatchPath -NoTypeInformation   
      If ($CreateMigrationBatch -eq $true) {  
           New-MigrationBatch -Name $BatchFileName -CSVData ([System.IO.File]::ReadAllBytes(“$BatchPath”)) –Local –TargetDatabase $DestinationDB -AllowUnknownColumnsInCsv $true