Office 365 vs. On-Premises Exchange Server

A large client is currently running Exchange 2010 and is evaluating moving to Office 365 vs upgrading to Exchange Server 2016. I talked with them about it and thought it would be useful to document it for future reference.

If you are a very small organization, then Office 365 is a slam dunk. It's going to perform better and be more cost effective than your could ever implement on your own. This is even before we consider the cost of the the consultants to get your on-premises Exchange up and running.

For mid-sized and large businesses there are more things to think about....

Cost

Direct cost is the first thing everyone wants to evaluate when considering Office 365. Your exact costs are going to vary depending on how you want to implement Exchange and which Office 365 plans you think are appropriate. So, I'm going to let you figure out the exact costs, but here are the things you need to consider:

On premises:

• Exchange Server licenses
• Exchange Server CALs (basic and enterprise if required)
• Hardware for the Exchange servers
• Storage for the Exchange servers
• Backup software and storage
• Antispam and antivirus solution

 Office 365:

• Monthly or yearly license (subscription-based)

High Availability

On-premises Exchange Server can be made highly available. This is done by configuring multiple Exchange servers in a database availability group (DAG) and by using load balancers. This configuration is well understood but definitely requires some planning and expertise to implement.  In most cases, hardware load balancers are used which are an additional cost.

Other high availability considerations for on-premises Exchange include redundant Internet connections, and alternate data centers. To provide a solution that is truly highly available, it takes a lot of redundant infrastructure.

Office 365 is highly available out of the box. You don't need to do anything, that's just the way it's designed. Your data is automatically located in multiple data centers in your region. If one data center fails, your data is still safe in another data center and you're still able to access it. This is one of the big advantages for Office 365, particularly for mid-sized organizations that might not have the resources to deploy office 365 across multiple data centers.

Authentication

On-premises Exchange Server uses Active Directory (AD) for authentication. When users are on a domain joined computer, authentication is performed automatically by Outlook. Externally, the AD username and password are used.

For Office 365, there are multiple ways that authentication can be configured. First lets talk about user accounts...

Very small organizations might choose to create user accounts in Office 365 manually. For these organizations, the users will sign in by using their email address and a password that is set in Office 365. This password could be different from the AD password on-premises and there is no synchronization of the passwords between AD and Office 365.

Larger organizations will use Azure AD Connect to synchronize user accounts with Office 365. This avoids the need to manually create the accounts. Azure AD Connect also has an option to synchronize passwords with Office 365. This simplifies password management for users. It also simplifies password resets when users call the help desk.

When passwords are synchronized with Office 365, all authentication is still handled by Office 365. There is no reliance on any on-premises infrastructure for the authentication process. So an on-premises service outage does not cause an Office 365 service outage.

Very large organizations may consider using Active Directory Federation Services (AD FS) for Office 365 authentication instead of password synchronization. When AD FS is implemented, Office 365 directs authentication requests to AD FS which then authenticates the request against AD. This is significantly more complex to configure than password synchronization but provides the following benefits:

• True single sign-on where workstation credentials are passed through for authentication just like on-premises Exchange.
• When an on-premises AD account is locked or disabled, this also applies to Office 365 authentication. Without this, disabling an Office 365 account is a separate process.

A key point to remember is that if your AD FS infrastructure is down, so is Office 365 authentication.

Account Lockout

For on-premises Exchange, the account lockout policies in AD are applied to authentication for Exchange mailboxes. The only exception is if a device such as a proxy is performing pre-authentication. If pre-authentication is performed then the device providing pre-authentication might have more restrictive policies than what is configured in AD.

When password synchronization is used with Office 365, there is a account lockout mechanism included in Office 365. After 10 failed attempts, an account is locked for 1 minute and if account lockout is triggered multiple times the lockout time is increased each time. This account lockout mechanism is not configurable.

When AD FS is used with Office 365, the account lockout policies in AD are used.

Mailbox and Archive Size

In on-premises Exchange, the maximum size of mailboxes and archive mailboxes is limited by the storage design. Or, more often by the cost of the storage being used. Using direct attached storage (DAS) with 7.2K SAS drives (rather than a SAN) is the most cost effective solution to provide large mailboxes and archives.

In Office 365, the maximum size of mailboxes and archive mailboxes is limited by the license assigned to the mailbox:

• Kiosk: 2 GB mailbox, no archive
• E1: 50 GB mailbox, 50 GB archive
• E3: 50 GB mailbox, unlimited archive
• E5: 50 GB mailbox, unlimited archive

Online Reading and Editing Attachments

In Exchange 2010, there was basic functionality for rendering attachments as web content in OWA. In Exchange 2016, this functionality is not included. Instead, you need to implement Office Online Server. Office Online Server is included with most volume licensing agreements for Exchange Server 2016.

Implementing Office Online Server is not terribly complex for a single server, but requires at least an additional virtual machine. If you want the functionality to be highly available you need to implement multiple servers and load balancing.

Office 365 includes the ability to read and edit attachments by default. No additional configuration is required.

Spam and Antivirus Filtering

For on-premises Exchange, you need to purchase additional products for spam and anti-virus filtering.

Office 365 licenses include Exchange Online Protection for anti-spam and virus protection at no additional cost.

Network Configuration

Most implementations of on-premises Exchange Server require you to implement proxies, load balancers, and firewalls. External access from the Internet to the Exchange servers also need to be configured.

For Office 365, you just need to ensure that clients on the internal network can communicate with Office 365 over the Internet.

Client Communication

Most on-premises deployments of Exchange server use Outlook in cached mode. However, because the internal network is reliable and fast, you can implement Outlook in online mode instead. This can be useful if you are concerned about the size of cached data on computers that are shared by multiple users.

Outlook clients must communicate with Office 365 servers over the Internet. This may result in higher Internet bandwidth usage. Generally, Outlook cached mode is used to minimize network utilization and provide better performance over networks with high latency.

Recent versions of Outlook cache only recent data in a mailbox rather than the entire mailbox. This mitigates concerns about large mailboxes being cached in local profiles. The amount of data cached is configurable.

Migration

Exchange Server 2016 can co-exist with Exchange Server 2010 or Exchange Server 2013. Migration is just a matter of moving mailboxes to Exchange Server 2016. There is almost no impact on users and the Outlook client is automatically redirected to the new mailbox location.

Office 365 can be integrated with Exchange Server 2010 or later by using hybrid mode. Once in hybrid mode, mailboxes can be migrated to Office 365 with minimal impact on users. Outlook is automatically reconfigured when mailboxes are moved.

Backup

Most common backup software can be used for backup with an appropriate agent on the Exchange servers. The restore functionality in the software varies but most can restore individual mailboxes or databases. This software is an extra cost.

There are third party options available for backing up Office 365 mailboxes, but there is no built-in functionality in Office 365 for this purpose.  However the same Single Item Recovery available in on-premises Exchange can be used in Office 365 to recover items for up to 30 days (default 14 days) after they are removed from deleted items.

Microsoft maintains multiple copies of mailbox data to avoid the need to restore from backup due to server error.

If you use E3 licenses, you can implement a hold on mailboxes to permanently retain messages received or sent from mailboxes. This held data is searchable for discovery purposes. Data holds can also be implemented for specific retention periods. Microsoft is positioning this type of hold to replace journaling and archiving solutions.

Management

For an on-premises Exchange deployment you need to perform the following management tasks:

• Monitor performance
• Monitor hardware for failures
• Update firmware
• Apply operating system updates
• Apply Exchange Server updates
• Manage mailboxes

On a system that is properly designed, there is minimal need to monitor performance (you built it with some head room), but there is an ongoing need to address issues such as failed indexes and large delivery queues. These issues are not relevant for Office 365 management.

As a cloud-based service, you are not responsible for any monitoring or updates in Office 365. Management of user mailboxes is still required.

New management tasks for Office 365 include:

• Managing directory synchronization
• Managing user licenses

Directory synchronization typically requires little management. There may be occasional troubleshooting, but it is rare. Generally once directory synchronization is configured, it just works.

Processes for creating new mailboxes do need to be modified to include assignment of Office 365 licenses. This can be scripted and run as a scheduled task if it is not required to be time sensitive.

If all users are migrated to Office 365 a local Exchange server is still maintained for management purposes. This is required because changes are made in the local AD and synchronized to Office 365. However, outages for this management server do not affect Office 365 users only management functionality. Microsoft provides a free license for this purpose.

Additional Features

Many Office 365 deployments begin as a replacement for on-premises Exchange. However, for mid-sized and large organizations, when you do a raw comparison of Exchange on-premises costs and Office 365 licensing costs, you may find that Exchange on-premises is actually a lower cost.

You need to consider that Office 365 includes more than just Exchange functionality. Depending on the licenses you select, there are various features. Also note, that you can mix and match licenses as required to give different features to different users to meet the needs of your organization.

For an accurate list of Office 365 plan features, see the Microsoft web site. However, here's a quick summary you can use to get a general idea of what's included with each license type:

• Kiosk:
• Exchange Online
• SharePoint Online
• E1: Kiosk features plus
• One Drive for Business (1TB)
• Skype for Business
• Yammer
• E3: E1 features plus
• Office 365 Proplus (desktop office suite)
• Azure Rights Management
• E5: E3 features plus
• Cloud PBX

These extra features provide more value than just Exchange online. I've been surprised at the number of larger organizations that have subscribed to Office 365 with E3 licenses as their preferred method of deploying Microsoft Office. Skype for Business is also a bit of a pain to implement on-premises and having that included with the E1 licenses is beneficial.

Modifying Mailbox Replication Latency Tolerance for Mailbox Moves

UPDATE: As of May 2024, a better option than this is to add the -Priority Highest option to the New-MoveRequest cmdlet.

Get-Mailbox -Arbitration | New-MoveRequest -TargetDatabase HybridDB -Priority Highest

============================

During the summer I was doing an Exchange 2013 to Exchange 2016 migration. All was going well except that mailbox moves would stall out and take a very long time. I didn't capture the logs at the time, but when looking at the logs, it indicated that the mailbox moves were stalling due to disk health and more specifically, disk latency.

To view the status of individual mailbox moves that are not completed, use the following command:

Get-MoveRequest | Where {$_.Status -ne "Completed"} | Get-MoveRequestStatistics

The default output for Get-MoveRequestStatistics only shows the DisplayName, StatusDetail,  TotalMailboxSize, TotalArchiveSize, and PercentComplete. However, there are some other useful properties in there for stalled moves such as:

• TotalStalledDueToContentIndexingDuration
• TotalStalledDueToMdbReplicationDuration
• TotalStalledDueToMailboxLockedDuration
• TotalStalledDueToReadThrottle
• TotalStalledDueToWriteThrottle
• TotalStalledDueToReadCpu
• TotalStalledDueToWriteCpu
• TotalStalledDueToReadUnknown
• TotalStalledDueToWriteUnknown
• DiagnosticInfo
• Report

In my case a slow mailbox move was showing stalling due to Content Indexing Duration (caused by slow disk) and Write Throttle (triggered by disk latency).

To make things easier I tend to dump everything into a variable so that I can look at them different ways and one by one. The example below collects all of the move request statistics in a single variable and then displays the all of the properties for the first move.

$moves = Get-MoveRequest | Where {$_.Status -ne "Completed"} | Get- MoveRequestStatistics
$moves[0] | Format-List

Now the preferred option is to figure out why your disk latency is high and try to resolve that. However, in this scenario, we needed to get the mailboxes moved because there were too many other things on the SAN to tinker with.

So, to speed up the mailbox move process, I tweaked the configuration file for the Mailbox Replication service to allow a higher level of latency before stalling out on the moves. This file is C:\Program Files\Microsoft\Exchange\v15\bin\MsExchangeMailboxReplication.exe.config.

At the end of this file there are a number of settings that control the service. For disk latency the settings are at the very end in the appSettings section. As shown below:

  <appSettings>  
   <!-- Mdb latency health threshold values in msec. Valid range is from 0-1000. -->  
   <add key="MdbFairUnhealthyLatencyThreshold" value="20"/>  
   <add key="MdbHealthyFairLatencyThreshold" value="10"/>  
   <!-- Maximum delay that WLM returns under "Fair" database resource health in msec. Valid range is from 0-60000 -->  
   <add key="MdbLatencyMaxDelay" value="60000"/>  
   <add key="LogEnabled" value="true" />  
   <add key="LogDirectoryPath" value="%ExchangeInstallDir%Logging\MailboxReplicationService\" />  
   <add key="LogFileAgeInDays" value="30" />  
   <add key="LogDirectorySizeLimit" value="100MB" />  
   <add key="LogFileSizeLimit" value="10MB" />  
   <add key="LogCacheSizeLimit" value="2MB" />  
   <add key="LogFlushIntervalInSeconds" value="60" />  
  </appSettings>  

The two settings for disk latency are MdbFairUnhealthyLatencyThreshold and MdbHealthyFairLatencyThreshold which have default values of 10ms and 20ms. In most cases those are reasonable defaults. I incrementally increased the values until the mailbox moves stopped stalling. My case was extreme and I ended up using values of 150ms and 75ms. The lower value of 75ms latency seemed to trigger the stalling.

When the config file is modified, you need to restart the Microsoft Exchange Mailbox Replication service to take effect. This will stop all mailbox moves that are in progress. They will start up again after the restart is complete but you may lose the progress that has already been made. For example, if 1.5 GB of a 2 GB mailbox has been moved, the whole mailbox move might start over again.

A couple more articles that could be helpful if you're looking at modifying this config file:

• Exchange 2013 workload management controls how the Mailbox Replication service uses system resources
• Speed up mailbox migration to Exchange 2013 - They're too slow!

KB3161949 Breaks Scan to Folder (and more...)

For small businesses, KB3161949 should not be a cause for concern. But for larger businesses with multiple subnets (don't need to be too large for that), KB3161949 might break multiple services that require NetBIOS over IP.

Your first thought is probably: Who the heck still uses NetBIOS over IP? I know that Windows doesn't need it anymore.

And, you're right. Windows (at least Vista and later) work just fine without NetBIOS over IP. However, there are a lot of devices with older SMB implementations that do require NetBIOS over IP. I ran into one Sharp MFP device today that requires it for scan to folder functionality.

Information about the update is published at: https://support.microsoft.com/en-us/kb/3161949

This update is a security update for WPAD. WPAD is a method for automatically configuring devices with a proxy server. However, as part of locking down this process, it does one critical thing. Here's the quote:

NETBIOS communication outside of the local subnet is hardened. Therefore, by default, some features that depend on NETBIOS (such as SMB over NETBIOS) will not work outside the local subnet.

So, communication on the local subnet works, but if the MFP and server hosting the share are on different subnets then it breaks.

Fortunate the fix is pretty quick, add a registry key and reboot the server hosting the share:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NetBT\Parameters
Value Name: AllowNBToInternet
Type: Dword
Value: 1

Another Microsoft page that describes the same update, but slightly different description: https://support.microsoft.com/en-us/kb/3165191

It's also worth noting that when troubleshooting this, the Security event log did not contain failed logon attempts for the user accounts. NetBIOS over IP was being dropped at the networking level and authentication was never attempted. 

Exchange Server Unable to Verify CRL

Just recently ran into another issue related to certificate revocation list (CRL) verification. This time it was an Exchange 2010 organization that had been fine when we initially installed the certificates, but now in the Exchange Management Console was showing the currently assigned certificate with a red X and an error message indicating that the CRL for the certificate could not be verified.

The certificate was still valid, but the Exchange server couldn't verify that it hadn't been revoked. No clients were affected by this issue. Viewing the certificate on a client accessing OWA showed as valid.

Like many organizations, this organization has a proxy between the internal network and the Internet. For the Exchange Servers to verify the CRL, they need to download it from the source specified in the certificate. This had been working, so, what changed?

It turns out that as part of troubleshooting connectivity to WSUS from the Exchange servers, the proxy configuration was removed. The connectivity for CRL verification is handled by the operating system. So, we needed to setup the proxy again.

The simplest way to configure the proxy for the operating system is to first configure the proxy settings in IE and them import them. After configuring IE proxy settings properly, use the following steps:

1. Open a command prompt.
2. Type netsh and press Enter.
3. At the netsh prompt, type winhttp and press Enter.
4. Type import proxy source = ie and press Enter.

For some other notes about Exchange Server and proxies, see: http://byronwright.blogspot.ca/2014/09/firewalls-and-proxy-for-exchange-hybrid.html

For some additional info about verifying certificates, see: http://byronwright.blogspot.ca/2016/10/expired-offline-root-ca-crl-causes.html

Expired Offline Root CA CRL Causes Warnings

Today at a client site I noticed an error in the System log of all the domain controllers:

Source: Kerberos-Key-Distribution-Center
Event ID: 29
Level: Warning
The Key Distribution Center (KDC) cannot find a suitable certificate to use for smart card logons....

 

This client isn't using smart cards, but there has been some certificate strangeness is the past. So, it's worth investigating.

The most common reason for this error is a missing or invalid domain controller certificate. If the organization has an internal CA then it should automatically get a domain controller certificate from that CA. If the organization does not have an internal CA, then you can consider the warning cosmetic and ignore it.

In this case, the organization does have an internal CA. In fact, they have an offline root CA and an enterprise issuing CA.

Each domain controller has a certificate that is issued from an LDAPoverSSL certificate template. This certificate ensures that all of the domain controllers are properly configured to respond to LDAPS queries from applications.

When I look at the LDAPoverSSL certificate on a couple of servers, they are not expired. And when I view them in the Certificates MMC snap-in they are identified as valid. So, all looks good

I also verified that these certificate did have the necessary options enabled for smart card authentication. In the Enhanced Key Usage field, the following were listed:

• KDC Authentication
• Smart Card Logon
• Server Authentication
• Client Authentication

The Event ID text mentions verifying the certificate by using certutil.exe. So, I find a couple of web sites that recommend running certutil -verify, but this requires you to have the certificate in a file. Instead, you can run the following command on the server containing the certificate you want to check:

certutil.exe -verifystore MY > cert.txt

This command verifies all of the certificates in the local certificate store of the computer. Note that you can also verify user certificates by adding the -user option, but computer certificates are check without any option. The output from the command is quite long and I'm dumping it to a text file for easier reading.

The output contains information for each certificate in the store. In my case, I identified the LDAPoverSSL certificate in the output and found the following at the end of the data:

The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613)
------------------------------------
Revocation check skipped -- server offline
Certificate is valid

Finally, a clue. So, I opened up the certificate and looked at the CRL Distribution Points field. This provided both an HTTP URL and an LDAP URL. When I accessed the HTTP URL, I was able to download and view the CRL without issue.

Certificate trust requires that the entire certification path is valid. So, next I looked at the issuing CA certificate in the chain. This certificate was also valid but pointed at a different CRL (same server, but different CRL file name for the offline root CA). When I downloaded this file I saw that Next Update field was two days ago. This corresponds within 30 minutes of when the warning began appearing in the event log.

To verify the validity of the certificate, the CRL for all certification authorities in the certification chain must be valid. Because the CRL from the offline root CA was expired, the certificate was not being trusted for all purposes. The fix is to update the CRL from the offline root.

We completed the following process to update the CRL:

1. Power on the offline root CA
2. Verify that the offline root CA updated the CRL at startup
3. Copied the new CRL from the offline root CA to the CRL distribution point
4. Published the new CRL to AD for availability via LDAP

To publish the CRL to Active Directory use the following command:

certutil -f -dspublish "CRL Path"

After this process was complete I verified the certificate again and did not get an error about skipping the revocation check.

Two important lessons:

1. The Certificates MMC snap-in does not check the CRL for a certificate. So, to check certificate validity, use certutil.exe.
2. If you have an offline root CA, you need to have a process in place to update the CRL from it.

Links with more information:

• You receive a Key Distribution Center "Event ID: 29" event message on a Windows Server 2008-based domain controller
• LDAP over SSL (LDAPS) Certificate
• Why do I see 'Unable to check revocation because the revocation server was offline'?