Netvanta UC Poor Quality Audio with NIC Teaming

I have one customer that uses Netvanta UC for their voicemail system. It runs on a virtual machine hosted on a Hyper-V cluster. That's a bit of an issue in itself because the software fails if the VM is migrated between hosts. To migrate it between hosts you need to reactivate it. Not cool.

For the past while, they've been having audio quality issues with this system and have investigated all points in the system, but none seemed to show any signs of an issue.

I'll let the customer describe it for you:

We have been plagued with garbled voicemail for a while now. Finally, we found some Adtran techs who took a hard look and found that when voicemail was hitting our NetvantaUC server sometimes (fairly often actually) up to 35% of the packets were out of order, causing a robotic and sometime impossible to understand recording. They also checked our PRI gateway and noted that there were no issues from the PRI (MTS) or leaving the gateway. Of course this left one switch in question, which we had already ruled out (no issues on the switch we could find).

This sort of left the VM host itself. I removed one nic from the team on Host 03 (where our NetvantaUC vm is hosted) and dedicated it to the NetvantaUC VM. For the past three days we have had no garbled voicemail and touch wood no phone issues at all (we had other phone issues daily, dropped calls, on hold and transfer occasionally not working etc).

So, it appears that there is some odd interaction between NIC teaming on Windows Server 2012 R2 and the Netvanta UC. At this point we're not sure if this is a general issue with VoIP software and NIC teaming or something specific to Netvanta UC.

The Total Data Received From the Remote Client Exceeded Allowed Maximum

Working with a large number of mailboxes is usually about the same a working with a small number of mailboxes, except that you need to include -ResultSize Unlimited in your Get-* cmdlets. However, I recently ran into the following error when getting a large list of mailboxes (approx 38K) with Get-Mailbox and piping them to Set-Mailbox:

Sending data to a remote command failed with the following error message: The total data received from the remote client exceeded allowed maximum. Allowed maximum is 524288000.

In the end this seemed to be a limitation of piping from Get-Mailbox to Set-Mailbox. For example, this would generate the error:

Get-Mailbox -ResultSize Unlimited | Set-Mailbox -SingleItemRecoveryEnabled $true

One option to work around this is to chunk up the work and do it by database instead:

Get-Mailbox -Database XXX -ResultSize Unlimited | Set-Mailbox -SingleItemRecoveryEnabled $true

However, my preferred workaround was this:

$mbx=Get-Mailbox -ResultSize Unlimited
foreach($m in $mbx) {Set-Mailbox $m -SingleItemRecoveryEnabled $true}

Using a foreach loop instead of piping seems to avoid the error when dealing with large data sets. The powershell.exe process for this task consumed about 2.5GB of memory.

List Disabled User Accounts with Mailboxes

I'm currently working with a large organization where they've lost track of mailbox removals when accounts are disabled. They keep disabled accounts for an extended period of time after users leave. So, some disabled accounts are years old and still have a mailbox.

We're going through and cleaning up right now. So, we wanted a list of disabled user accounts with an Exchange Mailbox. However, since resource mailboxes have a disabled user account, we need to exclude those. It's a nice easy one line of PowerShell:

Get-Mailbox -Filter {ExchangeUserAccountControl -eq "AccountDisabled" -and IsResource -eq $false} -ResultSize unlimited | export-csv C:\temp\DisabledWMailbox.csv

View Service Status for MTS

If you are having data connectivity issues on the MTS network this web site will show you the status of their services. Useful to figure out whether it's systemic or just you (or your customer).

• http://www.mts.ca/mts/support/service+status

Exchange 2010 Addressbook Errors with NetScaler 11.0

I'm working with a client that is upgrading the load balancing infrastructure that services their Exchange 2010 organization. They are implementing NetScaler SDX appliances that run NetScaler VPX virtual load balancers. Being as this is new infrastructure, version 11.0 (first release June 30, 2015, latest release Oct 8, 2015) is being implemented.

To test the new load balancing, we modified the hosts file on few computers to direct the load balanced names (outlook.domain.com/webmail.domain.com) to the IP address of the new load balancer. Initially, this seemed to be working fine, but we didn't do any specific testing.

After a bit, we noticed that we often got the following error in Outlook when accessing the address book:

The connection to Microsoft Exchange is unavailable. Outlook must be online or connected to complete this action.

This error was reproducible after not accessing the address book for a few minutes. In addition, when looking at the connection status, we would see failures talking to the directory. Connectivity to mail generally seemed fine, but also had occasional failures.

Connection Status with directory access failures

When you research this problem, you will find lots of references to incorrect network timeouts closing connections before the application is ready. This is exactly what we seemed to be having. However, we adjusted every possible setting on the NetScaler to allow connections to last up to two hours. No matter what we adjusted, the same problem remained.

Other network components were the same as before. So, it seemed the problem had to be the new NetScaler. Clients still using the existing load balancer did not have any issues.

After adjusting everything we could think of, we called Citrix support. In conjunction with support, the network team did some packet traces and confirmed that resets were being performed on the RPC connections for the address book.

The fix ended up being using version 10.5 (latest release Sept 2015) instead of version 11.0 for the VPX instance. As soon as version 10.5 was implemented the mysterious reset issue was resolved and the response time for the directory lookups dropped significantly.

Connection Status after implementing version 10.5

Citrix is continuing to evaluate the issue and I'll update when I have more information.

Update (Nov 10/15): Apparently NetScaler 11.0 has issues with Outlook Anywhere also: http://discussions.citrix.com/topic/368750-netscaler-11-6210-outlook-anywhere-broken/page-2

Update (Nov 13/15): Running version 10.5 we've still had no issues. It has been submitted as a bug with Citrix and we're waiting for a update. Hopefully the next revision will fix it, but they are definitely aware it's an issue.

Update (Nov 27/15): Citrix did some testing with the network folks again today with Version 11.0 (I'm assuming a new build) and still not working properly. We are back to Version 10.5.

Resources:

• NetScaler Release Software Dates: http://support.citrix.com/article/CTX121840
• http://blogs.technet.com/b/david231/archive/2015/03/30/for-exchange-2010-and-2013-do-this-before-calling-microsoft.aspx
• https://discussions.citrix.com/topic/348704-exchange-2010-rpc-timeout-issue/
• https://social.technet.microsoft.com/Forums/exchange/en-US/e7c5ab7c-7ccc-4f4a-ac40-31831aea2d81/outlook-time-out-setting?forum=exchange2010
• https://masterxen.wordpress.com/2015/08/11/exchange-rpc-timeouts-when-using-citrix-netscaler/

Query Mailbox by LegacyExchangeDN

In a large organization with mailbox quotas, you will see an ongoing number of events indicating that users are at the limit of their quotas. In the event log you will see:

Event Source: MSExchangeIS
Event ID: 8528
The mailbox for /o=ExchangeOrg/ou=AdministrativeGroup/cn=Recipients/cn=UserIdentifier has exceeded the maximum mailbox size. This mailbox cannot send or receive messages. Incoming messages to this mailbox are returned to sender.  The mailbox owner should be notified about the condition of the mailbox as soon as possible.

The overall identifier for the mailbox is the LegacyExchangeDN attribute for the user mailbox. Most of the time, you can look at this value and quickly figure out which user is having the issue. If it's a current user, you can wait or contact them. In a large environment, you may not personally know the users, and you can investigate whether this is a disabled account that needs to be removed.

Today I was looking at one of these events and saw that the final UserIdentifier portion of the LegacyExchangeDN attribute was a long GUID number. To find the actual user account I needed to query the mailbox with a filter for LegacyExchangeDN.

Get-Mailbox -Filter {LegacyExchangeDN -eq " /o=ExchangeOrg/ou=AdministrativeGroup/cn=Recipients/cn=UserIdentifier"}

Please note that using the -Filter parameter is much more efficient that doing Get-Mailbox and piping it to Where-Object. Using the -Filter parameter is almost instant whereas piping Get-Mailbox to Where-Object can take several minutes.

StartDagServerMaintenance.ps1 Error with 2 Database Copies

When you have a DAG with only two nodes, the StartDagServerMaintenance.ps1 script is smart enough to recognize that you can't have 2 additional copies of the database. However, if you have more than 2 DAG members, then the script complains that it can't run because the local database copy is required for redundancy.

I ran into this at a client where we created a database in the DR site for running test cmdlets with SCOM. There are two copies of the database in the DR site. Other databases have two copies in the main site and one copy in the DR site.

You can override the default behaviour with the switch OverrideMinimumTwoCopies.

Example:

StartDagServerMaintenance.ps1 -Server DAGnode1 -OverrideMinimumTwoCopies

Getting Status Totals in PowerShell

I was doing an Exchange migration on the weekend and had a large number of move requests. Being the slightly OCD computer person as most of us are, I wanted to see how things were progressing occasionally. However, I didn't want to be dumping stuff into spreadsheets and be counting items.

After few false starts, I ran across the Group-Object cmdlet. For me, this cmdlet is in the same category as Measure-Object. I've never had a need in the past. Now I'm happy to have it.

My solution:

Get-MoveRequest | Group-Object -Property Status

The results looked like this:

Synchronizing Remote IP Ranges Across Recieve Connectors

Exchange 2010 and later do a nice job of providing high availability with database availability groups (DAGs) and load balancing. However, one configuration detail doesn't automatically synchronize between multiple Exchange servers, and that is receive connectors.

If you create receive connectors for relaying output from printers or scanners then the connector you create is unique on each server. That's fine if you are pointing the devices at individual Exchange servers but to have high availability, you need multiple load balanced servers with the same configuration. To do this, you need to create the same receive connectors on each server.

During intial setup creating 2 or 4 receive connectors with the same settings for authentication and such isn't too big a deal. The item that's a pain is the remote IP ranges that are allowed to use the receive connector. Many organizations have a large list of individual IP addresses that are allowed to use the receive connector.

If there is a list of 50 individual IP addresses, it's a long process to enter once, let alone multiple times. Not to mention, there is the risk of administrators adding or removing IP addresses to the list on one server, but not others, or typos.

Here is a quick and easy way to synchronize the remote IP ranges from a receive connector on one server to another.

$Source=Get-RecieveConnector Server\ConnectorName
$Destination=Get-ReceiveConnector Server\ConnectorName
Set-ReceiveConnector $Destination -RemoteIPRanges $Source.RemoteIPRanges

In the example above:

• $Source is the connector you are copying the remote IP ranges from
• $Destination is the connector your are copying the remote IP ranges to

It is not necessary to use a variable for the destination receive connector. I did that to clarify what is happening in the Set-ReceiveConnector command. You could put the identity of the destination receive connector directly in the Set-ReceiveConnector command.

If you need to do this on a regular basis, you can put the receive connector into an array and use a foreach loop to set them all in a script like this:

$Source=Get-RecieveConnector Server\ConnectorName
$Destination=Get-ReceiveConnector | Where { $_.name -like "Relay*"}

ForEach ($d in $Destination) {
    Set-ReceiveConnector $d -RemoteIPRanges $Source.RemoteIPRanges
}

Can't Create Domain Certificate with Custom Template in IIS Manager

I was teaching an informal class this week on certificates and Active Directory Certificate Services (AD CS). When a class is informal, you don't necessarily create detailed lab activities ahead of time. It's more like you have a general flow you want to do. Consequently, something just don't work as you expect.

We used IIS Manager to create a domain certificate from the internal CA that we has setup. That part worked fine based on the Web Server template that exists automatically after installing AD CS.

After that I thought this would be a great way to demonstrate superceding a template. Turns out I was wrong. The wizard in IIS Manager will only request certificates based on the Web Server template. If you take that one away, the wizard for creating a domain certificate doesn't work. We were using Windows Server 2012 R2, but this behavior is also consistent in Windows 2008/2008 R2 (IIS 7/7.5)

The work around is to use the Certificates snap-in to create the certificate from AD CS.

Reference:

• http://blogs.technet.com/b/instan/archive/2011/04/06/why-can-t-i-see-any-certificate-templates-when-creating-a-certificate-request-within-the-iis-7-x-mmc.aspx

iOS 9 Increases Security Requirements

Apple recently released iOS 9 and this is important for Exchange administrators. Many implementations of Exchange 2010 are implemented on Windows Server 2008 R2 which does not meet the security requirements for iOS 9 to connect.

Apps on iOS 9 or later and OS X 10.11 or later use App Transport Security. App Transport Security defines minimum security standards that must be met for a network connection to be successful. The default behavior requires:

• TLS version 1.2
• Perfect forward secrecy
• SHA256 or better for certificate signature
• 2048 bit or better RSA key (or 256 bit or better ECC key)

If your Exchange server does not meet the App Transport Security requirements then the connection is dropped by iOS or OS X. This applies to web services. So, it will affect OWA, ActiveSync, and EWS (Outlook for Mac).

We are just starting to see iOS 9 installed by clients and are in the process of ensuring that servers meet the requirements.

NOTE: As of right now (Sept 20/15), iOS 9 does not appear to be enforcing the above requirements for internal applications but does for third party applications. We are continuing to update all Exchange servers to meet these requirements as they are best practices for security and because they could affect OS X 10.11 when released.

TLS 1.2

TLS 1.2 has been around since 2008 but is not enabled by default on Windows Server 2008 R2. Windows Server 2012 and later have TLS 1.2 enabled by default. Windows Server 2008 does not include TLS 1.1 or TLS 1.2.

To enable TLS 1.2 on Windows Server 2008 R2, you need to create a registry key and reboot the server:

1. Open Regedit.
2. Browse to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols.
3. In Protocols, create a new Key named TLS 1.2.
4. In TLS 1.2, create a new Key named Server.
5. In Server, create a new DWORD (32-bit) Value:
• Name: DisabledByDefault
• Value: 0
6. In Server, create a new DWORD (32-bit) Value:
• Name: Enabled
• Value: 1
7. Reboot the server.

 

Perfect Forward Secrecy

Perfect Forward Secrecy ensures that each connections uses a difference encryption key. Windows Server 2008 R2 support the use Perfect Forward Secrecy, but again, it's not enabled by default. There are a limited number of ciphers that use Perfect Forward Secrecy. It is a matter of enabling these ciphers and also changing the order of preference so that they are preferred.

I've done quite a bit of searching and there is no good guide for manually enabling Perfect Forward Secrecy. However, there are several scripts you can download and even a graphical tool.

My personal favorite is the graphical tool IIS Crypto from Nartac Software because it provides an easy way to see exactly which protocols and ciphers are being modified. Press the Best Practices button to view the optimal security settings and then Apply and reboot the server. This will enable TLS 1.2, disable SSL 2 and SSL 3 and enable Perfect Forward Secrecy in one step.

Script to enable Perfect Forward Secrecy and more:

•  https://www.hass.de/content/setup-your-iis-ssl-perfect-forward-secrecy-and-tls-12

SHA256

SHA is the algorithm used to sign the digital certificates installed on your Exchange server. Many older certificates use SHA1 which is relatively insecure. You can update your certificate by regenerating the certificate. Most provides allow you to regenerate the keys at no cost. During this process you can select to use SHA256 and then import in Exchange Server.

2048 Bit Key

Several years ago, certification authorities started requiring a minimum key length of 2048 bits. So, it is unlikely that this is an issue for your certificate. However, if you had a certificate with a long lifetime, you can regenerate it with a 2048 bit key to bring it up to requirements.

Testing Your Configuration

The simplest way to verify the SSL/TLS configuration on your server is to use the SSL test from Qualys SSL Labs. This web site connects to your server and runs it through a variety of tests before providing a report on the SSL configuration. It's the fastest way to find out whether specific versions of SSL/TLS and ciphers are enabled or disabled.

Windows Server 2012 and Windows Server 2012 R2

This blog post has focused on Windows Server 2008 R2 because there is a lot of it out there running Exchange 2010. However, Windows Server 2012 and Windows Server 2012 R2 are used for many implementations of Exchange 2013.

Windows Server 2012 and Windows Server 2012 R2 support TLS 1.2 by default. If you have installed update KB2919355 released in April 2014, then Windows Server 2012 R2 uses Perfect Forward Secrecy. For Windows Server 2012, you need to configure the cipher protocols to enable Perfect Forward Secrecy just as was required in Windows Server 2008 R2. Again, using a script or IIS Crypto is the preferred method of enabling and reordering the necessary ciphers.

References

Links for reference:

• App Transport Security Technote: https://developer.apple.com/library/prerelease/mac/technotes/App-Transport-Security-Technote/index.html
• TLS versions: https://en.wikipedia.org/wiki/Transport_Layer_Security
• TLS/SSL Settings Reference: https://technet.microsoft.com/en-us/library/dn786418.aspx
• Qualys SSL Labs test: https://www.ssllabs.com/ssltest/index.html
• How to restrict the use of certain cryptographic algorithms and protocols in Schannel.dll: https://support.microsoft.com/en-us/kb/245030 
• Nartac IIS Crypto download: https://www.nartac.com/Products/IISCrypto/Download
• Update adds new TLS cipher suites and changes cipher suite priorities in Windows 8.1 and Windows Server 2012 R2: https://support.microsoft.com/en-us/kb/2929781

Cannot achieve Exchange Server authentication

In a single server Exchange environment, you can get away with some misconfiguration because the Exchange servers don't need to communicate with each other. Recently I was working with a larger client that made a common configuration error on receive connectors. This client had the following configuration:

• multi-role DAG members in datacenter #1
• multi-role DAG members in datacenter #2
• all mailbox databases are replicated between the two datacenters
• 1 public folder database in each datacenter

Message delivery was working properly between servers in datacenter #1. Only passive database copies were in datacenter #2. However, there were issues with public folder replication. Some public folders were not replicating properly from datacenter #1 to datacenter #2.

Looking at the queues for servers in datacenter #2, you could see that the queue between AD sites had a large number of messages. The following error was present on the queue:

451 4.4.0 Primary target IP address responded with: "451 5.7.3 Cannot achieve Exchange Server authentication"

The servers had been configured with a receive connector for relaying with only anonymous authentication. This had been done to allow various applications to relay messages through as necessary. On the Network tab of the relay connector, Receive mail from remote servers that have the IP addresses was configured with a range of IP addresses that included the Exchange servers.

When an Exchange server receives an inbound request for SMTP, it uses the source IP address of the request to select the appropriate receive connector. The preferred receive connector has the most specific IP address range that includes the source IP address. A single IP address is preferred over a range and small range is preferred over a larger range. If connectivity fails due to authentication errors no other connectors are used and communication fails.

Normally when Exchange servers send SMTP messages between themselves, they use the Default receive connector which is configured for Exchange Server authentication. The Default receive connector has a source IP range of 0.0.0.0-255.255.255.255. When the server is first installed, this ensures that communication works properly. However, any receive connector you create is more specific than this and will be preferred over the Default receive connector.

That is what happened in this case. The relay connector that included the IP addresses of the Exchange servers was preferred over the Default receive connector that had the proper authentication settings. This is a common error.

The fix was to add the specific IP addresses of the Exchange servers to the Default receive connector on each server. This made the Default receive connector more specific than the relay connector.Once this was done the error disappeared and queues delivered messages properly.

One question is, why was only public folder replication affected and not mail flow within the AD site?

Within an AD site, the Hub Transport role on a server communicates directly with the Mailbox role on another server with the mailbox database by using RPC. So, the Default receive connector is not required for message delivery within the AD site.

In this case, the only inter-site SMTP messages were the public folder replication messages. No databases were mounted in Datacenter #2 . So, no email messages were sent there via SMTP. Database replication does not use SMTP.

The mail flow configuration in this scenario should be relevant for Exchange 2007, Exchange 2010, and Exchange 2013. However, the error message may vary slightly.

Hung at Couldn't Complete Updates Undoing Changes

Setting up a new VM for a client today and ran into a new issue. Well new to me. Fortunately others have seen this before.

In a brand new Windows Server 2012 R2 VM (provided by Dell), I downloaded all of the available updates from Windows Update and started the installation. After a long period of time, the VM rebooted and I got the following message:

We couldn't complete the updates
Undoing changes
Don't turn off your computer

I'm a reasonably patient guy, but after 40 minutes of this I decided that waiting longer wasn't going to fix this. The first round of searching suggested that I disable Secure Boot for the VM. Apparently others had good luck with this:

• http://blog.powerbiz.net.au/fixes/we-couldnt-complete-the-updates-undoing-changes-error-on-windows-server-2012-r2/

Unfortunately I restarted the VM and Windows Server 2012 R2 hung in exactly the same spot again. Although I only waited about 10 minutes this time before declaring it hung. So, what's next.

There is at least one Windows Update that expects Bitlocker to be installed for Gen 2 VMs. You don't need to encrypt anything but the feature needs to be installed. On my system it was Security Update KB2920189 that seemed to cause the issue.

You might be able to install BitLocker by booting the VM into safemode. However, there is a better way. On the Hyper-V host, you can use Server Manager to add the feature to the VM while it is turned off.

To see the details with beautiful screenshots, check out this link:

• http://andyparkes.co.uk/blog/index.php/2015/03/25/adding-roles-and-features-to-an-offline-virtual-machine/

After installing the BitLocker feature in the VM, some updated completed installing quickly and I could log in. Most of the updates failed and I had to run Windows Update again, but all was good.

Framed Route Configuration for MTS Internet

I recently found out that you can have multiple IPs on a MTS (Manitoba Telecom Services) Internet connection. I haven't investigated this for a long time and this was a pleasant surprise when migrating a client from a more expensive Internet connection to MTS after MTS upgraded the connectivity in their relatively remote building.

Adding additional IPs with MTS is done by purchasing a framed route. The minimum size is 6 IP addresses for $9.95/month. Pretty good if you need an extra IP or two as our client did. The description of the service is here:

• https://www.mts.ca/mts/business/internet/features/framed+route

We have the framed route up and running for the customer and it worked very well once it was configured. What we ran into was vague documentation and a support person that wasn't familiar with it and messed up the configuration on the first attempt (a second tech was very helpful and got it right).

MTS provides the following document to clients:

• https://www.mts.ca/mts/support/internet/high+speed/framed+route+customer+equipment+configuration+guide

The document provided by MTS pretty much just indicates to add an IP from your framed route range to your computer or firewall and it will all start working. However, that is not the case. Your modem from MTS needs to be configured to support the framed route. It's not hard to do, but there is no documentation available, and it's not widely understood by the support staff.

We did the install for a Pace modem. So, the screenshots below show what needs to be configured for that specific modem type.

After you have connected to your Pace modem, go to Settings > Broadband > Link Configuration.

At the bottom of this page, in the Supplementary Network section, select Enable to add an additional network.

Notice that the interface doesn't ask you for your network, it only asks for the address you want to use as the default gateway (router address) and the subnet mask. The Pace box identifies the network from this information. So, you need to determine ahead of time which IP address in your framed route that you want to use as gateway. This means if you purchased a framed route of 6 IP addresses, only 5 are usable because the gateway uses one IP address.

You should also select the Auto Firewall Open option to automatically forward all Internet traffic to hosts inside the Pace box. If you don't, you'll need to either forward specific ports in the Pace firewall, or configure additional DMZ zones in the Pace box. In our case, we were configuring a firewall behind the Pace box and wanted all traffic to be forwarded.

Optimize Network Connectivity for Office 365

Microsoft has just released a nice video on optimizing network connectivity for Office 365:

• How to optimize your network connectivity and performance with Office 365 and Azure

A quick summary of the video:

A lot of connectivity to Office 365 is not in your control. However, you should have an understanding of connectivity to the data center to your tenant. In some cases, MS has worked with ISPs to optimize connectivity in cases where there was obviously bad routing. For example traffic being routed unnecessarily over trans-oceanic links.

If you have high latency to Office 365 you can use tools such as PSping to look at connectivity. PSping performs a connection to a service at a port number rather than using ICMP packets as regular ping does. You can also use Network Monitor to identify some issues.

PSping is free from Sysinternals:

• https://technet.microsoft.com/en-us/sysinternals/jj729731.aspx

It's important to know that DNS lookups are based on geographic location.  For example outlook.office365.com will resolve to a different data center depending on location. If users are accessing through a corporate VPN they may be accessing inefficiently.

Creating Shared Mailbox in a Hybrid Deployment

The first thing to be aware of when creating shared mailboxes in a hybrid deployment is security. Sharing mailboxes between on-premises and O365 is not supported. So, if a group of people need to share a mailbox then their mailboxes all need to be on-premises or all in O365.

On-Premises Shared Mailboxes

Creating an on-premises shared mailbox is pretty straight forward. Create the shared mailbox in the on-premises Exchange and it all works.

In Exchange 2013, shared mailboxes are explicitly listed as a recipient type in the Exchange admin center (EAC). You can create and manage the shared mailboxes there.

In Exchange 2010, shared mailboxes are not part of the Exchange Management Console (EMC). You need to create the shared mailbox by using the New-Mailbox cmdlet in the Exchange Management Shell (EMS). For example:

New-Mailbox HelpDesk -shared -UserPrincipalName HelpDesk@MyDomain.com

After creating the shared mailbox in Exchange 2010, you need to give users permission to access it. Assign Full Mailbox permissions to let users manage the contents of the mailbox. You may also want to give SendAs permissions depending on your scenario.

Office 365 Share Mailboxes

In Office 365, the web-based management interface provides the same option to create shared mailboxes as Exchange 2013 does. However, in a hybrid environment, you can't create the shared mailboxes directly in Office 365.

If you create the share mailbox directly in Office 365 there is no Active Directory reference to the shared mailbox on-premises. This prevents Outlook from properly adding the shared mailboxes because autodiscover does not work properly. In a hybrid environment, autodiscover is directed to the on-premises Exchange organization and won't be able to direct Outlook to the correct location of the shared mailbox because there is no information in Active Directory about the shared mailbox in the on-premises AD.


In a hybrid environment, you should perform the following steps instead:

1. Create a Remote Mailbox in Office 365 from the on-premises Exchange organization.
2. Run Dirsync (or wait for several hours).
3. In Office 365, convert the mailbox to a shared mailbox. Available when the recipient is selected as seen in the screenshot to the right.
   
4. In Office 365, configure Full Access and SendAs permissions to the shared mailbox as required.

It's a bit more of a hassle to create a shared mailbox in Office 365 for a hybrid environment, but it does work!

When you create the shared mailbox directly in Office 365, you'll see the following symptoms:

• Shared mailboxes are not automatically added to Outlook.
• If you attempt to add the shared mailbox to Outlook manually in the properties of the Exchange account, then Outlook will continually prompt for credentials and hang.

Script for Exchange 2013 Message Tracking

Exchange Server 2010 had a graphical utility for analyzing message tracking logs. Unfortunately, this tool was removed from Exchange Server 2013. Instead in Exchange Server 2013, you have only the Get-MessageTrackingLog cmdlet.

The Get-MessageTrackingLog cmdlet is a pain in the butt for a few reasons:

• You need to memorize the syntax. Most of it is pretty straight forward, but you need to remember the correct parameters for searching by sender, recipient, or subject.
• It only searches the local server by default. Without specifying servers, it only searches the local Exchange server that you're running the tool on. In a lot of cases, you need to see information from all your servers to track it down.

While working on a message delivery problem this week, I wrote up a short script help with simple message tracking based on time, sender, recipient, or message subject. The script is as follows:

 Write-Host "Current Date/Time: $(Get-Date)"  
 $StartTime = Read-Host "Start time for search"  
 $EndTime = Read-Host "End time for search"  
 $SearchType = Read-Host "Search for (S)ender, (R)ecipient, (M)essage subject, or display (A)ll"  
 Switch ($SearchType) {  
   'S' {  
            $Sender = Read-Host "Sender"  
            Get-ExchangeServer | where {$_.isHubTransportServer -eq $true -or $_.isMailboxServer -eq $true} | Get-MessageTrackingLog -Start $StartTime -End $EndTime -Sender $Sender |Select-Object Timestamp,ServerHostname,ClientHostname,Source,EventId,Sender,Recipients,MessageSubject | Out-GridView }  
   'R' {  
            $Recipient = Read-Host "Recipient"  
            Get-ExchangeServer | where {$_.isHubTransportServer -eq $true -or $_.isMailboxServer -eq $true} | Get-MessageTrackingLog -Start $StartTime -End $EndTime -Recipient $Recipient |Select-Object Timestamp,ServerHostname,ClientHostname,Source,EventId,Sender,Recipients,MessageSubject | Out-GridView }  
   'M' {  
            $MessageSubject = Read-Host "Message subject (performs partial matches)"  
            Get-ExchangeServer | where {$_.isHubTransportServer -eq $true -or $_.isMailboxServer -eq $true} | Get-MessageTrackingLog -Start $StartTime -End $EndTime -MessageSubject $MessageSubject |Select-Object Timestamp,ServerHostname,ClientHostname,Source,EventId,Sender,Recipients,MessageSubject | Out-GridView }  
   'A' {  
            Get-ExchangeServer | where {$_.isHubTransportServer -eq $true -or $_.isMailboxServer -eq $true} | Get-MessageTrackingLog -Start $StartTime -End $EndTime |Select-Object Timestamp,ServerHostname,ClientHostname,Source,EventId,Sender,Recipients,MessageSubject | Out-GridView }  
   default {Write-Host "Invalid Option - Run Script Again"}  
 }  

Here is how the script works:

1. The current date/time are displayed. This shows you the date/time syntax to use for entering time in the next steps.
2. You are prompted for the time to start searching the logs.
3. You are prompted for the time to stop searching the logs.
4. You are prompted for the type of search you want to do: sender, recipient, message subject, or display all.
5. The switch command uses the $SearchType variable to run a specific code block. The command varies depending on the option, but in general, it prompts for the required information and then runs the query based on it.
6. Results are displayed by using Out-Gridview. This allows you to sort based on columns.

Notes:

• This code is used to identify and generate a list of all Exchange servers with message tracking logs which is then piped to the Get-MessageTrackingLog cmdlet.
  

  Get-ExchangeServer | where {$_.isHubTransportServer -eq $true -or $_.isMailboxServer -eq $true}  

• When you search by message subject, it returns all results that include the snippet of text. This can make it hard to track down the specific message that you're looking for sometimes. For example, searching for "text" will include messages with "text" in the subject, but also "context","textbook", etc.
• Only 1000 results are returned by the Get-MessageTrackingLog cmdlet. It's possible to override this, but if your query is returning more than 1000 results, you should probably be refining your query.
• Times from Exchange 2007 servers seem off. I was testing in a Mixed 2013 and 2007 environment and the content coming back from the 2007 environment had timestamps outside the range I queried. I have not yet had time to investigate, but be aware of this when sorting results based on time.

KB 3062157 Breaks Single Role Exchange 2013 Servers

It appears that there may be problems, in at least some cases, when you apply KB 3062157 to servers running Exchange Server 2013 SP1 or later. It will break web services such as OWA, ActiveSync, EWS, and ECP.  The KB is meant to address vulnerabilities described in Microsoft Security bulletin MS15-064.

• https://support.microsoft.com/en-us/kb/3062157

The good news is that this problem appears to affect only  single role servers and not multi-role servers. Since most deployments have multi-role servers that will limit the impact.

If you deploy this update and experience problems, removing the update should resolve the issue and get the server functional again.


This update is also included as part of CU9 and doesn't appear to have the same issues when installed as part of CU9. A commenter on the blog below had problems with KB3062157, uninstalled that update, and installed CU9 without issue.

• http://exchangeserverpro.com/ms15-064-important-security-update-for-exchange-server-2013/

Automatic Activation for Hyper-V VMs

When you use Windows Server 2012 R2 Datacenter Edition as the host operating system for Hyper-V, you can implement an unlimited number of virtual machines using that same license. This means that you can buy a single Windows Server 2012 R2 Datacenter Edition license for a single physical server and run an unlimited number of guests using Windows Server 2012 R2 as the operating system.

If you have multiple Hyper-V hosts, it can be a pain to keep track of the Windows Server 2012 R2 keys for the guest VMs. Instead of using typical activation methods, you can use Automatic Virtual Machine Activation (AVMA).

AVMA activates a guest VM against the Hyper-V host instead of typical Microsoft activation methods. This means that the VM can be completely isolated without access to the Internet or other network and still be activated.

You can use AVMA guest VMs running:

• Windows Server 2012 R2 Essentials
• Windows Server 2012 R2 Standard
• Windows Server 2012 R2 Datacenter

For each guest operating system you enter in a generic AVMA license key. The generic license keys are:

• Datacenter - Y4TGP-NPTV9-HTC2H-7MGQ3-DV4TW
• Standard - DBGBW-NPF86-BJVTX-K3WKJ-MTB6V
• Essentials - K2XGM-NMBT3-2R6Q8-WF2FK-P36R2

To enter the generic AVMA key, you can use any method that you typically use to enter a license key. During installation, you can use an unattend.xml file. After installation, you can run the following command:

slmgr /ipk AVMAlicensekey

To monitor AVMA licensing requests on the Hyper-V host, look for Event ID 12310. On the guest VMs look for Event ID 12309.

UPDATE: Microsoft link that also includes AVMA keys for Windows Server 2016

• https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-R2-and-2012/dn303421(v=ws.11)

Perc H700 Connector Confusion

Throwing this one out there in case anyone hits a similar issue. This one confused the heck out of me but was simple in the end.

We have a customer with a Dell T310 server with an H200 RAID adapter. The performance on the H200 is pretty poor. In part because the drive cache is disabled by default. But, search around and you'll see no one has much nice to say about the H200.

To improve performance for this server, we ordered an H700i card from Dell. We ordered from Dell rather than aftermarket to avoid issues with compatibility. And so that it would be supported.

There are several sets of instructions out there for doing this upgrade. Here is the one I thought was the best:

• https://practicalsbs.wordpress.com/2014/08/06/upgrading-dell-perc-h200-raid-card-to-h700/

We went through the guide, installed the card, installed drivers, and then went to connect the drive backplane to the card. Uh oh, wrong connector type.

You can see above that the new card has a two prong connector that does not match the mini SAS connector (shown below) of the cables that shipped with the card. The cable already in the server also had this type of connector.

While I'm comfortable with server hardware, I'm by no means an expert on all different connection types. So, at this point I assume that there is a connector type that I'm unaware of and we need either a different card or different cables.

My rep confirmed that this is in fact the correct card. Next step is calling support. I sent the pictures to support and he wasn't sure what was up either. While on the phone as the support rep was searching, I tried searching for SAS connector types and nothing matched what I was seeing on the card. It wasn't making sense. This can't possibly be correct.

At this point, I tried gently pulling on the plastic part of the connector and it came out (see below). The plastic was a spacer put in the connector for shipping. After removing the plastic plug the connector fit the cable properly. It seems obvious in retrospect, but it didn't at the time.

One final note about this card. We ordered the model with 1 GB non-volatile RAM. I assumed that this meant no battery was required. In fact the card uses DRAM for operations because it is faster than NVRAM. The battery is still required to move data from DRAM to NVRAM when a power outage occurs.