Throttling Hyper-V Replication Traffic

We have a client with two physical locations and a 100 Mbps link between them. For disaster recovery purposes, we are using Hyper-V replication between the two sites.

Recently we made a disk change to one of the VMs that forced us to delete and recreate replication for one of the VMs. Unfortunately it was the VM with 650 GB of data. Because replicating that amount of data in the best case will take about 13 hours over the WAN link, we need to control the replication and prevent it from interfering with normal business.

Hyper-V does not have any built in functionality to control bandwidth for replication traffic. Fortunately Windows Server 2012 R2 has quality of service (QoS) functionality built into the operating system.

In my case, the receiving server is using port 443 to receive the data. On the source server, vmms.exe is replicating the data to the destination server. I created policy that limited traffic to 50 Mbps from vmms.exe to port 443 with the following command:

New-NetQosPolicy "VM Replication" -IPPortMatchCondition 443 -AppPathNameMatchCondition vmms.exe -ThrottleRateActionBitsPerSecond 50000000

Be aware that there are other options that you can use to refine the policy. For example, you can specify destination IP, source port, destination port, or protocol (TCP or UDP). There is lots of flexibility to be as precise as you need to be.

After creating a policy, you can manage it by using:

• Set-NetQosPolicy
• Remove-NetQosPolicy

Microsoft documentation for New-NetQosPolicy:

• https://technet.microsoft.com/en-us/library/hh967468%28v=wps.630%29.aspx

Exchange 2013 Is Filling My C: Drive

So, Exchange 2013 does a few things differently than previous versions of Exchange. If you are not careful about how you allocate your disk space, you'll end up constantly wondering why the C: drive is filling up. And if your monitoring isn't up to snuff, you'll notice C: drive is filling up because mail flow almost stops due to back pressure.

If you are using direct attached storage for your C: drive as Microsoft suggests, then you likely have a fairly large C: drive. Something like 300GB or more. If you have a C: drive that large you're likely OK and don't need to worry about it. On the other hand, if you have your Exchange server using a SAN and you tried to keep your C: drive to 80GB because that SAN space is expensive, you will have issues.

There are three common things can use up the C: drive:

• Internet Information Services (IIS) logs
• Exchange Server diagnostic logs
• Transport queues

IIS Logs
Exchange Server has a variety of web-based services such as Outlook Web App (OWA) and Exchange Web Services (EWS). IIS hosts these web-based services and generates logs for them each day. Unfortunately IIS does not have an option to delete log files after a certain number of days. This is a problem that has existed for Exchange 2010 and Exchange 2007 also.

In most cases, the issue isn't so much the size of log files generated each day. Instead, it's the fact that the log files build up over time and begin to take up and significant amount of space.

You can move the IIS logs off of the C: drive by modifying the log settings for the Default Web Site. However, that still leaves you with a constantly building collection of log files. What we have started to do is delete all log files older than 14 days by using a scheduled task. This link provides instructions:

• http://byronwright.blogspot.ca/2015/01/script-to-remove-old-iis-logs.html

Exchange Server Diagnostic Logs
Exchange Server 2013 generates significantly more diagnostic logs than Exchange Server 2010. For Exchange 2010 Microsoft suggested 1.2GB available on D: drive. Now 30GB is recommended for C: drive and a big chunk of this is diagnostic logs. The logs are automatically purged after 14 days, but they are a significant amount of space. One of our clients with about 1700 users has 20GB of diagnostic logs.

If you don't allocate enough space originally on your server, there is no method for moving the logs to an alternate drive. However, you can use a junction point in the file system (created with mklink.exe) to redirect the folder to an alternate location.  Alternatively expanding C: to support the diagnostic logs is also good.

If you can't have a large C: drive when you install your server, consider installing Exchange Server on a separate drive. The diagnostic logs are kept in the installation folder:

• [install location]\Microsoft\Exchange Server\v15\Logging

Transport Queues
In Exchange 2013 the transport queues are significantly larger than Exchange 2010. This is due mostly to the new SafetyNet feature. This feature keeps a copy of mail messages for 2 day to help when recovering from disasters. As you can guess, this can end up a significant amount of data. One of our clients had transport queues growing to about 60GB per server.

Again, you could install Exchange initially on a separate drive to allow for the size of the mail.que file. However, if Exchange is installed to the C: drive, you can do a few things to mange mail.que.

First, you can move the transport queues to an alternate location. Microsoft provides a set of directions for this:

• https://technet.microsoft.com/en-us/library/bb125177%28v=exchg.150%29.aspx

Secondly, you can manage Safety Net to reduce the amount of data that is cached to less than 2 days if you deem that appropriate for your environment. You can do this from Exchange Administrative Center (EAC) at Mail flow > Receive connectors > More options > Organization transport settings > Safety Net > Safety Net hold time. You can also use the Set-TransportConfig cmdlet with the -SafetyNetHoldTime parameter.

For more information about Safety Net, see:

• https://technet.microsoft.com/en-us/library/jj657495%28v=exchg.150%29.aspx

Set a New Internal Transport Certificate

Normally, when we configure clients running Exchange 2013 with a valid certificate for web services, we also apply it to SMTP. This allows opportunistic TLS to be performed and secure email delivery. When you apply the certificate, it will ask:

• Overwrite the existing default SMTP certificate?

We normally say yes and our valid/trusted certificate is configured as the "internal transport certificate". This is all good.

Recently the certificate on an Exchange 2013 server was replaced and when the new certificate was applied, it was not configured as the default/internal transport certificate. This caused the following error when I attempted to remove the expired certificate:

 A special Rpc error occurs on server XXXXXXX: The internal transport certificate cannot be removed because that would cause the Microsoft Exchange Transport service to stop. To replace the internal transport certificate, create a new certificate. The new certificate will automatically become the internal transport certificate. You can then remove the existing certificate.

This error is suggesting that we use New-ExchangeCertificate to create a new self-signed certificate. That is a perfectly valid technical solution. The only thing we've noticed is that customers tend to ignore all certificates except the trusted one that they've installed. Then the internal certificate expires and causes some problem with mail delivery. Specifically, we've seen delivery errors between Edge Transport servers and internal Exchange servers.

I want to set our new valid certificate which is bound to IIS and SMTP as the default SMTP certificate. To do this I needed to use the console because the web interface won't let me reapply the same certificate. Here is the process.

1. Use Get-ExchangeCertificate to identify the thumbprint of the certificate you want to be default. This is a big long ugly number, do yourself a favor and copy it to clipboard instead of trying to type it.
2. Use Enable-ExchangeCertificate -Thumbprint XXXXXXX -Services 'iis,smtp' to assign the services.
3. Answer [Y] Yes when prompted to overwrite the default SMTP certificate.

After changing the default SMTP certificate, I could delete the expired certificate without error in Exchange Admin Center.

High Disk Utilization on SBS 2011

High disk activity on SBS 2008 and SBS 2011 is an ongoing concern for our clients. Just yesterday, we had an SBS 2011 server with so much disk activity that it slowed down the Hyper-V host so that all VMs were not performing properly.

To diagnose this issue, use Resource Monitor to view the Queue Length. When there is a problem the Queue length for a drive will be high and sustained. The % Active Time will also be high. These indicate that the disk is busy and has a backlog of work to perform.

It's important to note that the actual disk I/O may not be that high. In our case, the Disk I/O was less than 10 MBps which is far less than the disk system is capable of. The Disk I/O can be low but the disks busy when many small operations are performed.

The fix for this is almost always recreating the SBS Monitoring database. The SBS Monitoring database grows over time and eventually becomes a performance issue. We don't use any historical information in that database. So, removing it and recreating it is a good solution for us. It's a 10 minute fix that lasts for about a year before needing to be done again.

Microsoft instructions for how to recreate the SBS Monitoring database are here:

• http://blogs.technet.com/b/sbs/archive/2011/08/22/how-to-recreate-the-sbsmonitoring-database.aspx

You can verify that the SBS Monitoring database is the issue by stopping theSQL Server (SBSMONITORING) service. After stopping that service, you should see disk activity slow down significantly.

On a physical host, the SBS Monitoring database will cause high Disk I/O on the C: drive. On a hyper-V host, the SBS Monitoring database will cause high Disk I/O on the physical drive that is hosting the C: drive VHD for SBS. Monitoring Disk I/O inside of a virtual machine will not be accurate.

Office 365 Deleted Item Retention

Microsoft recently announced that deleted item retention in Office 365 will now default to indefinite. Until now, items in Deleted Items were removed after 30 days. This was announced here:

• http://blogs.office.com/2015/02/20/extended-email-retention-deleted-items-office-365/

Microsoft indicates in this blog that it's implemented based on ignoring Deleted Items if the policy name is the default name. I just migrated my mailbox to O365 and it appears that while items directly in Deleted Items are no longer removed after 30 days, items in subfolders are still removed.

In my newly migrated mailbox I had some old subfolders that had been removed and were in Deleted Items. There is no longer any content in those folders but all of the items directly in Deleted Items have been retained.

ActiveSync Settings for Office 365

Our office email system has been running in hybrid mode for the last year or so. That was mostly for us to do testing related to client projects. This weekend we moved all of our accounts to Office 365.

Autodiscover worked well for the Outlook clients, but not for the phones. For our Android phones we removed the existing account and readded the account. My Galaxy S4 did not do failover in autodiscover properly from our on-premises autodiscover to Office 365 autodiscover. So, I ended up manually configuring the ActiveSync account.

If you are manually configuring the ActiveSync account for Office 365, the server name is:

• outlook.office365.com