Thursday, September 14, 2017

Getting Details Error Messages for Mailbox Moves

In Office 365 or Exchange Server 2013/2016, you can use the administration console to view information about migration batches. To find out information about failing moves, you can view the details of the migration batch and then view the report for individual mailboxes. When you view the report for a mailbox a text file is downloaded for viewing.

The report provides detailed information about how much data has been downloaded. Also, if there are errors, they are contained in the report. Unfortunately sometimes the errors are pretty generic. For example, one error I got recently was:
Transient error TimeoutErrorTransientException has occurred. The system will retry (200/1300).
Instructions on how to review the report:
Since the error was happening often, we needed to get more information. Fortunately that detail is available, but not in that report. Instead, you need to use Windows PowerShell to view the move request statistics. If you are moving the mailbox to Office 365, you need to use a PowerShell prompt connected to Exchange online.

$stats = Get-MoveRequest user@domain.com | Get-MoveRequestStatistics -IncludeReport
$moveErr = $stats.Report.Entries | Where-Object {$_.Type -eq "Error"}

This leaves you will an array named $moveErr that contains all of the errors. You can view each error by specifying an individual array item.

$moveErr[1]

In my case, I got this more detailed information in the Failure property:

TimeoutErrorTransientException: The call to 'https://hybridserver/EWS/mrsproxy.svc
hybridserver (15.1.1034.26 caps:07FD6FFFBF5FFFFFCB07FFFF)' timed out. Error
details: The request channel timed out while waiting for a reply after 00:00:30.9527011.
Increase the timeout value passed to the call to Request or increase the SendTimeout
value on the Binding. The time allotted to this operation may have been a portion of a
longer timeout. --> The request operation did not complete within the allotted timeout of
00:00:50. The time allotted to this operation may have been a portion of a longer
timeout. --> The request channel timed out while waiting for a reply after
00:00:30.9527011. Increase the timeout value passed to the call to Request or increase
the SendTimeout value on the Binding. The time allotted to this operation may have been a
portion of a longer timeout. --> The request operation did not complete within the
allotted timeout of 00:00:50. The time allotted to this operation may have been a portion
of a longer timeout.

That error information gave me enough information to start looking at timeout settings for the MRS proxy service.

Tuesday, September 12, 2017

Using Let's Encrypt Certificates for Exchange Server

Have you ever fantasized about using free SSL/TLS certificates for Exchange Server? If so, then this blog post is for you.

I’ve always hated the cost associated with SSL/TLS certificates. For what seemed like a pretty basic service some of the certificate authorities (CAs) were charging hundreds or thousands of dollars. You could always set up your own CA, but that didn’t work well with random clients on the Internet because they won’t trust certificates generated by your CA.

At the end of 2015, there was a game changing development. Let’s Encrypt started giving away SSL/TLS certificates for free. At the time, the certificates were only for a single name. So, without SAN support, not good for Exchange Server. However, now there is support for SAN/UCC certificates. And, in 2018 they are planning to support wildcard certificates.

What’s the Catch?

The certificates are free. There is no catch there. But, they do have a short lifetime of 90 days. The short lifetime is to ensure that compromised certificates are not available for an extended period of time. Because of the short lifetime, it is strongly recommended that you automate certificate renewal.

Note: This blog post only shows the manual steps for obtaining a certificate. I'll put up another one showing automation.

The process for generating and renewing a certificate is a bit complex. But, once the initial process is defined, it’s pretty easy to work with.

Unlike a typical CA, Let’s Encrypt does not provide a web site to manage your certificate requests. Instead you need client software that communicates with the Let’s Encrypt servers. Since I already work with Windows PowerShell on a regular basis, I like the ACMESharp module that provides PowerShell cmdlets for working with Let’s Encrypt.

Installing the ACMESharp Module

The ACMESharp module is available in the PowerShell Gallery. To download and install modules from the PowerShell Gallery, you use the Install-Module cmdlet that is part of the PowerShellGet module. The PowerShellGet modules is included as part of the Windows Management Framework 5 (part of Windows 10 and Windows Server 2016).

If you are not using Windows 10 or Windows Server 2016, you can download and install WMF 5 or a standalone MSI installer for PowerShellGet here:

After you have PowerShellGet, installed run the command:
Install-Module AcmeSharp

When you run this command, you might be prompted to install NuGet. If you are prompted, say yes to install it. NuGet is provides the functionality to obtain packages from the PowerShell Gallery. The PowerShellGet cmdlets use NuGet.

You might also be prompted that the repository PSGallery is untrusted. This is the PowerShell Gallery that you want to download files from. So, say yes to trust PSGallery.



Connecting to Let’s Encrypt


The first step after installing the ACMESharp module is creating a local data store for the ACMESharp client. The data store is used by the client for secure storage of requests and keys.
To create the local data store run the following command:
Initialize-ACMEVault

Then, to create an account with Let’s Encrypt, run the following command:
New-ACMERegistration -Contacts mailto:youremail@yourdomain.com -AcceptTos

Validating DNS Names

Let’s Encrypt requires you to verify ownership of each DNS name that you want to include in a certificate. Each DNS name is referred to as an identifier. For a SAN certificate, you will generate 2 or more identifiers then specify the identifiers when you create the certificate.

You can validate an identifier in three ways:
  • Dns - You need to create a TXT record in DNS that is verified by Let’s Encrypt.
  • HTTP - You need to place a file on your web server that is verified by Let’s Encrypt.
  • TLS-SNI - You need to place an SSL/TLS certificate on your web server that is verified by Let’s Encrypt.
In the projects I work on, I typically do not have access to the main company web server/site, but do have access to create DNS records. So, I use DNS validation.

To create a new identifier:
New-ACMEIdentifier -dns server.domain.com -alias idAlias
You should include an alias each time you create an identifier. If you don’t create an alias, there is no easy way to refer to the identifier in later steps. If you forget to create an alias, just create a another new identifier with the same DNS name and include the alias.

In my example, the DNS name has four parts because I was testing by using a subdomain. In most cases, the DNS name will have only three parts.


Next you need to specify how the identifier will be verified. When you do this the cmdlet reports back the proof you need to provide. For HTTP verification, it identifies the file name. For DNS verification, it identifies the TXT record that needs to be created.

The command to start verifying the identifier is:
Complete-ACMEChallenge idAlias -ChallengeType dns-01 -Handler manual
Use the alias of the identifier to specify which identifier is being verified.

The manual handler identifies that you are specifying the challenge type. There are other automated handlers that automatically create the response for the HTTP-based challenges. The automated handlers are specific to different web servers.

The challenge type dns-01 identifies that you will create a TXT record in DNS. Note that dns-01 must be in lowercase. 


After you have created the DNS record that corresponds to the challenge, then you submit the challenge. When you submit the challenge, Let’s Encrypt verifies it.

To submit a challenge:
Submit-ACMEChallenge idAlias -ChallengeType dns-01

When you submit the challenge, you need to specify the alias of the identifier and the challenge type.
The validation may or may not complete immediately. You can verify the status of the validation with the following command:
(Update-ACMEIdentifier idAlias -ChallengeType dns-01).challenges
The Update-ACMEIdentifier cmdlet queries the status of the identifier from the Let’s Encrypt servers. The challenges property contains the challenges generated when the identifier was created. The status for the dns-01 challenge will change to valid when validation is complete.



Creating a Certificate

After you have validated all of the identifiers that will be included in your certificate, you can generate the certificate request. When you generate the certificate request, you need to specify the identifiers to include and a new alias for the certificate.

To generate the certificate request:
New-ACMECertificate idAlias -generate -AlternativeIdentifierRefs idAlias1,idAlias2 -Alias certAlias
The initial alias identifier that you provide is the subject for the certificate. This name is also added to the subject alternative names automatically. You don’t need to repeat it as an alternative identifier.
The -AlternativeIdentifierRefs parameter is used to identify additional identifiers that are included in the certificate. All identifiers used here need to be validated.



To submit the certificate request:
Submit-ACMECertificate certAlias
The certificate alias used here is the alias that was set when running the New-ACMECertificate cmdlet.


To export the completed certificate as a pfx file that includes the certificate and private key:
Get-ACMECertificate certAlias -ExportPkcs12 filename.pfx -CertificatePassword “password
The -ExportPkcs12 parameter can be given a filename or a full path. If there are spaces in either one, you need to put quotes around it.


When I first ran the Get-ACMECertificate cmdlet to export the pfx file, I got an error:
Issuer certificate hasn’t been resolved.

This error was caused by the computer with ACMESharp not having the necessary intermediate certificate for the Let’s Encrypt CA. After I installing the X3 intermediate certificate, it worked without issue.


If necessary, you can get the X3 intermediate certificate here:
After you have the pfx file, you can import it and assign to Exchange Server using the normal Exchange Management cmdlets.

Friday, September 1, 2017

Remove Proxy Address from Office 365 User

I ran into an issue today where I needed to remove a proxy address from a cloud-based administrative user in Office 365 that was unlicensed. This user had a proxy address that was conflicting with a proxy address that was being synced with Azure AD Connect for another user account.

The cloud user was originally created as byron@OnPremDomain.com and renamed to be byron@CloudDom.onmicrosoft.com. When this was done, the original address (byron@OnPremDomain.com) was kept as a proxy address. You could view both addresses when using Get-MsolUser. This address caused a synchronization error for an on-premises user named byron@OnPremDomain.com.

To resolve this error, I need to remove byron@OnPremDomain.com from the list of proxy addresses. However, you can't do this with Set-MsolUser. The mechanism for managing proxy addresses in Office 365 is Set-Mailbox. But, without a license, there is no mailbox for the user account.

The solution is to add a license temporarily:
  1. Add a license for byron@CloudDom.onmicrosoft.com which creates a mailbox.
  2. Use Set-Mailbox -EmailAddresses to remove the incorrect proxy address.
  3. Verify Get-MsolUser shows only the correct proxyaddresses.


Monday, August 28, 2017

AD Synchronization Error When Adding Exchange 2016

When I implement hybrid mode for an organization, we typically implement Exchange Server 2016 to be the long term hybrid server. This provides the most recent Exchange Server version for management.

Today when I was installing Exchange Server 2016 into an Exchange 2010 organization we started getting directory synchronization errors for some system mailboxes (four of them). This occurred after I ran /PrepareAD, but before the remainder of Exchange Server 2016 was installed.

SystemMailbox{xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx}@XXXXX.com

Unable to update this object in Azure Active Directory, because the attribute [Username], is not valid. Update the value in your local directory services.
We only noticed these errors because Office 365 sent an Identity Synchronization Error Report that listed them.

When I looked in Synchronization Service Manager to see the details in Azure AD Connect, the following errors were there in the export to the Office 365 tenant:
I did some searching around and found a few articles that talked about manually updating attributes for these objects. However, when I looked at these objects the data didn't match what those articles were talking about.

Eventually I found a posting in an MS discussion forum that said to just wait until the install was finished. Apparently /PrepareAD creates the objects, but their configuration is not complete until the rest of Exchange Server 2016 is installed.

Sure enough, after the Exchange Server 2016 install was finished the synchronization errors went away.

Posting in MS discussion forum for reference (see answer from Ytsejamer1):
I did also refresh the connector schema as suggested in this post. However, I did that after running /PrepareAD and before Exchange Server 2016 was installed. I can't say for sure whether this step is required.




 

Monday, August 21, 2017

Updating SIP Addresses in Skype for Business

When you migrate to Office 365, the preferred configuration is to have user email addresses and UPNs the same. Having a single identity makes it easier for users to understand.

If you are implementing Skype for Business in Office 365, it will take the UPN of the user as the Skype address. Again, keeping a single identity is good.

However, if you have an on-premises implementation of Skype for Business, then the Skype identity is configured in the attribute msRTCSIP-PrimaryUserAddress.  This attribute contains a SIP (session initiation protocol) address that looks like an email address but with “sip:” at the start. For example: “sip:user@contoso.com”.

The SIP addresses defined in your on-premises Skype for Business may or may not match the email addresses of the users. You need to verify whether the addresses match. If the SIP address does not match the email address, it is easy to change.

On the Skype Server run the following PowerShell command:
Set-CsUser -Identity userUPN -SipAddress “sip:emailaddress
When you update the SIP address for UserA there are two considerations:
  • UserA will be signed out of Skype within about 1 minute. After UserA is signed out, they need to sign in again by using their new SIP address.
  • Any users with UserA as a contact are not updated immediately. These users need to sign-out and sign-in again for the contact to be updated and properly view presence information. However, this only works if the address book as been updated.
Address book updates are the big gotcha in SIP address updates. By default, Skype for Business clients use a cached address book that only updates once per day. Even if you force address book updates on the server, the new address information is not retrieved until the next day unless you manually delete the cached address book files from the client to force a reload.

To address this problem, you can switch clients to use an online address book. Clients using the online address book query the Skype for Business server each time they search the address book. When using the online address book, changes show up within a few minutes.

Address book look ups are controlled by client policies. By default, there is one policy named Global. You can update this policy to use online address book resolution with the following command:
Set-CsClientPolicy -Identity Global -AddressBookAvailability WebSearchOnly
Alternatively, you can create a new policy and assign that to users incrementally:
New-CsClientPolicy -Identity OnlyWebAB -AddressBookAvailability WebSearchOnly
Grant-CsClientPolicy -Identity UserName -PolicyName OnlyWebAB
The new client policy takes effect the next time the user signs in. To increase the likelihood that users have the new policy, you should configure it and then wait until the next day before making large scale changes.

Some useful links:

Sunday, August 13, 2017

Scripting Complexity vs. Simplicity

Note: Most of my blog posts are technical items that relate to performing a specific task or fixing a specific error. This one is more of an opinion piece. So, if my arguments sound reasonable, take the advice. If you disagree, by all means, go your own way.

I am a big believer that I need to understand the details of any script I run in a production environment. Unless that script is from Microsoft and provided to perform a specific task (such as migrating public folders), I will go through a script line by line to verify I understand it. This even applies to my own scripts. If I haven’t used a script for several months, I’ll review it before I use it again to make sure I know exactly what it’s doing.

I expect that most system administrators operate with the same requirement to understand the scripts they are running. At least I hope they do. I don’t want anyone blindly running a script I created without understanding the script and knowing what it will do in their environment.

So, from my perspective, running a script is very different from running a program/utility. A program is distributed as an executable and you simply need to trust the developer. I don’t need to trust a script writer if I can understand what the script is doing. The writer just saved me the effort of creating the script myself. It’s much faster to review and verify an existing script than it is to create one.

Simple is good, right?

If you need to understand a script, then simple is good. However, we often attempt want a script to be resilient to user error. That is, we want the script to do things like:
  • Not accept invalid values
  • Not make incorrect/invalid changes to objects
  • Warn when mistakes are about to be made
A more user-friendly script is good, but it becomes more complex and more difficult to interpret. So, does that mean complexity is good?

Know your audience

The balance of simple (but you need to know what your doing) and complex (with lots of error checking) depends on your audience. If the audience needs to understand the details of your script, then simplicity is better. If the audience does not care about the details inside the script then simplicity is less important. For example, if the help desk staff in your organization run standardized scripts to complete some specific tasks then user friendly is more important than simplicity. The help desk staff will be running the script without reviewing the contents or understanding how it works. So, if they give an invalid value, you need to account for that in your script. It’s worthwhile to make it user friendly to prevent errors in the production environment.

The customers I work with typically will look at my scripts before running them. So, these scripts need to be relatively short and understandable. The documentation that I provide for running these scripts also needs to be short and understandable. In this case simplicity is most important for the audience. For example, if there are several tasks that need to be accomplished such as updating email addresses, UPNs, etc. I will create separate scripts for each task. This makes each individual script easier to understand and document.

Striking a balance

I tend to work with customers that have varying knowledge levels and needs. Some of them would run a script without looking and others want a detailed understanding how the scripts work. I want my scripts to work in both scenarios. So, I need to put in some of the user-friendly bits like prompting for values when required and some basic checking for invalid values, but I don’t try to account for every possible error. To account for every possible error would be too much complexity.

Because I can’t (or won’t) put in tons of error checking, one thing I often do is display a confirmation on screen before performing an action. For example, I have a script to remove email addresses for a specified domain. Before script performs the action, it displays on screen the pattern being searched for and provides an email address from the first mailbox as an example of what is about to be removed. The script also displays the number of mailboxes that are being modified. Displaying this information allows the user to provide a final sanity check before approving removal from all mailboxes.

To make scripts more understandable, I use lots of comments that describe what each section of a script is doing.  This is useful for customers that want to review the script. It’s also useful for me when I want to review the script contents or modify the script. Embedding comment-based help in a script makes it easier to use, but detailed comments in the script are better for understanding how it works.

If you don’t get the right balance then your scripts won’t get used. That help desk person will think your script generates errors all the time and ignore it. The administrators you gave complex scripts to will build their own because they can’t understand yours.

Friday, June 30, 2017

June 2017 Security Update Breaks Outlook Search

The June 2017 security update for Windows causes problems for search in Outlook clients using cached mode.  Uses will see incomplete search results and may be notified that:
Search results may be incomplete because items are still being indexed.
This issue affects Windows 7, Windows 8.1, and Windows 10. It also affects all versions of Outlook.


Microsoft has fixed this issue with a new update released on June 27th. You can download the update here:
These fixes are for retail, OEM, and volume licensed versions of Outlook. If your Outlook is included with an Office 365 version of Microsoft Office, then updates are already available and should be installed automatically. If they are not installed automatically performed the following steps:
  1.  In Outlook, click the File tab, and click Office Account.
  2. Click Update Options and click Update Now.

To review Microsoft's documentation on this issue, see Issue #5 on this page:

Wednesday, June 28, 2017

Exchange 2010 OWA failure

Had a call about an older Exchange 2010 server this morning. Users were having problems working with Outlook Web App.

I saw a high volume of errors in that Application event log. This error was referring to both autodiscover and '/EWS/Exchange.asmx' and was appearing multiple times per minute.

Event ID: 3
Source: System.ServiceModel 3.0.0.0
Category: WebHost
WebHost failed to process a request.
 Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/32001227
 Exception: System.ServiceModel.ServiceActivationException: The service '/Autodiscover/autodiscover.xml' cannot be activated due to an exception during compilation.  The exception message is: This collection already contains an address with scheme http.  There can be at most one address per scheme in this collection.
Parameter name: item. ---> System.ArgumentException: This collection already contains an address with scheme http.  There can be at most one address per scheme in this collection.
There was also this error about OWA.

Event ID: 108
Source: MSExchange OWA
Category: Configuration
Outlook Web App couldn't connect Exchange Web Services due to a configuration error. Response code = "null, webException.Status = SendFailure".
The errors in the log began at about 12:15am when someone had done a server reboot. So, something in that reboot triggered the issue. My first thought was updates, but no updates had been applied recently.

After a bit of research, it turned out to be the bindings for the default web site causing the issue. At some point, someone had added additonal http and https bindings to the default web site. When I removed the additional http and https bindings everything started properly and the errors no longer appeared.

I am guessing that someone added those bindings a long time ago, but they didn't start causing the issue until IIS was restarted as part of the reboot.

Tuesday, June 27, 2017

For Winnipeg IT Pros

For anyone from the Winnipeg IT Pros group reading the blog. Here are the PowerShell links that I promised to post.

PowerShell learning resources:
Slide deck from my presentation:

Friday, June 23, 2017

Errors on Public Folder Migration

As I was doing a public folder migration today, I got a couple of errors that took me some time to resolve. These are caused by mail enabled public folders migrated from Exchange 2003. You will see these errors when you run Get-MailPublicFolder on Exchange 2010. Some of these errors will show up in the public folder migration logs when migrating to Exchange 2016. So, I prefer to clean these up first to ensure migration is successful.


Error #1

WARNING: The object domain.com/Microsoft Exchange System Objects/PF Name has been corrupted, and it's in an inconsistent state. The following validation errors happened:
WARNING: Could not convert property OnPremisesObjectGuid to type Guid. Byte array for GUID must be exactly 16 bytes long.
My best guess is that this property is left over from Exchange 2003 (or maybe earlier). The quick fix is to disable mail for the public folder and then mail-enable it again. However, when you do so, verify the email addresses before and after.

Error #2


WARNING: The object domain.com/Microsoft Exchange System Objects/PF Name has been corrupted, and it's in an inconsistent state. The following validation errors happened:
WARNING: Property expression "PF Name" isn't valid. Valid values are: Strings formed with characters from A to Z (uppercase or lowercase), digits from 0 to 9, !, #, $, %, &, ', *, +, -, /, =, ?, ^, _, `, {, |, } or ~. One or more periods may be embedded in an alias, but each period should be preceded and followed by at least one of the other characters. Unicode characters from U+00A1 to U+00FF are also valid in an alias, but they will be mapped to a best-fit US-ASCII string in the e-mail address, which is generated from such an alias.

This error is most commonly caused by a space in the Alias property. Update this property to remove spaces and the error should be gone.

Saturday, June 17, 2017

Multiple Moderation Approval Requests

I recently did a migration from Exchange 2010 to Exchange 2016 where the client uses a high volume of moderated messaging. There were over 100 transport rules that did message moderation of some sort. The initial deployment consisted of Exchange 2010 SP3 RU17 and Exchange 2016 CU4.

Deployment of Exchange 2016 into the Exchange 2010 environment didn't seem to have any effect. However, after we directed the internal namespace to Exchange 2016 for proxying, the approvals generated by the transport rules when whacky (yep that's the technical term).

Here is the process we saw:
  1. Message requiring moderation sent.
  2. Approval request sent to moderator.
  3. Moderator approves request
  4. Approval request sent to moderator
  5. Moderator approves request
  6. Repeat request and approval process a few more or a lot more times.
This process was happening even though we had not moved any mailboxes to Exchange 2016 yet. 

When searching, there were very few references to this issue on the Internet or support forums. However, there were a few suggestions that were consistent:
  • Ensure arbitration mailboxes are moved to Exchange 2016 (one of these stores the messages until they are approved).
  • Delete and recreate rules.
  • Move moderators mailbox to Exchange 2016.
  • Restart transport services.
All of these things were done but we still had issues. However, when both mailboxes were on Exchange 2016, the approvals on messages seemed to only happen twice. This was better than the random number from before.

I reviewed the message tracking logs for errors and didn't see any. In the logs, each time an approval was received, the message was released for delivery, but then promptly moderated again. However, the second attempt to approve worked.

All of my initial testing was done using inbound messages. So, I tried some scenarios with both mailboxes on Exchange 2016, and these were my results:
  • Inbound messages routing through Exchange 2010 first - Two approvals required
  • Outbound messages routing out through Exchange 2010 - One approval only
  • Inbound messages routing through Exchange 2016 only - One approval only
Based on these results, we can see that Exchange 2010 coexistence definitely plays a role, because when Exchange 2010 is not part of the inbound routing the issue doesn't occur. This at least provided confidence that after migration was complete the issue would not be persist.

The other item that needed to be addressed was the Exchange 2016 CU4. Microsoft releases updates in matched sets and Exchange 2016 CU4 was one step behind Exchange 2010 SP3 RU17. In the hope that having both at the same update level would fix it, we applied Exchange 2016 CU 5. We did the update to CU 5 but there was not change.

Final testing indicated that any message that entered the Exchange organization through Exchange 2010 was subject to double approval. This happened if the message came in from external or was generated by a mailbox in Exchange 2010. The location of the moderator mailbox did not make a difference.

So, to minimize the issue:
  • Move all inbound message routing to Exchange 2016 sooner rather than later. This includes Internet mail and applications that send messages to be moderated.
  • Move mailboxes that generate the most messages to be moderated first. Once the source is in Exchange 2016 the problem is mediated.
I should also note that there was a red herring in the application event log. We saw this error:
Event ID 1051, MSExchange Extensibility
Warning, MExRuntime
 Agent 'Approval Processing Agent' caused an unhandled exception 'SmtpResponseException: 250 2.1.5 APPROVAL.ApprovalRequestUpdated; approval request updated successfully' while handling event 'OnCreatedMessage'
However, review of the logs indicated that the error was present before the issue appeared. So it appears to be noise rather than useful information.

Thursday, June 15, 2017

Firefox - Exchange 2016 and NS_ERROR_NET_INADEQUATE_SECURITY

I'm working on an Exchange 2016 migration project and it was all looking good. We setup the load balancer and verified it worked for OWA and Outlook. Things were good.

Today we did the switch over and updated the DNS records to route all of the client traffic through Exchange 2016. IE and Edge were fine, but Firefox gave an error:
NS_ERROR_NET_INADEQUATE_SECURITY
Protocol issues for web browsers, certificates, and web servers can be tricky. It's hard to track it down with generic error messages like this. So, as an alternative to figuring out the details, I used IIS Crypto from Nartac with the best practices settings on the Exchange 2016 servers. After those were applied all was good. Just apply the settings and reboot.

For more about using the free IIS Crypto to see:
UPDATE: Here is an ongoing thread from the Exchange Technet forums. Seems to be known issue for Exchange 2016 at this point.

Tuesday, June 6, 2017

SourceMailboxAlreadyBeingMovedTransientException

Today while moving a mailbox from on-premises to Office 365 in a hybrid environment, I got the following error:
Transient error SourceMailboxAlreadyBeingMovedTransientException has occurred. The system will retry (5/620).
This error occurs when a previous move attempt did not get cleaned up properly. From a bit of reading, this should timeout and fix itself after about 2 hours. However, since I didn't want to wait that long, I did the following that got it going again.
  • IISReset.exe to restart the web services
  • Restart the Microsoft Exchange Mailbox Replication service
It is possible that only one of those two items was required, but I was more concerned about getting the move going than recording exact details.

Monday, May 29, 2017

Firefox Error Accessing Office 365

Microsoft is having a certificate trust issue today for Firefox when accessing Office 365 services. I haven't bothered digging into the details, but it's something to do with trust checking by using OCSP (Online Certificate Status Protocol).

The error you see is "SEC_ERROR_OCSP_INVALID_SIGNING_CERT" and looks like this:
Other browsers are not affected. So, you work around this by using Chrome, IE, or Edge.

If you prefer to continue using Firefox, you can modify the setting for certificate verification with the following steps:
  1. In the address bar, type about:config and press Enter.
  2. Click the I accept the risk button.
  3. Scroll down to security.ssl.enable_ocsp_stapling and double-click it to set the value to false.
Given that other browsers don't use this type of security, it's fairly safe to disable.

Friday, May 12, 2017

Everyone Dynamic Group with Office 365 Hybrid

Many organizations have configured a dynamic distribution group that includes all users with mailboxes. This Everyone group is used for sending out company notifications and the like. Because this group is dynamic, it's automatically updated as new mailboxes are added and removed.

When you implement hybrid mode with Office 365, the dynamic group will include on-premises mailboxes but not cloud mailboxes. This is because cloud mailboxes are a different recipient type that is not included. Cloud mailboxes are MailUser recipients (an AD user with an email address but no mailbox).

One way to fix this is to modify the dynamic distribution group to include MailUser recipients. The screenshot below has enabled Users with external e-mail addresses.


This will include users with cloud mailboxes. This is the recipient filter for the dynamic distribution group:
((((RecipientType -eq 'UserMailbox') -or (RecipientType -eq 'MailUser'))) -and (-not(Name -like 'SystemMailbox{*')) -and (-not(Name -like 'CAS_{*')) -and (-not(RecipientTypeDetailsValue -eq 'MailboxPlan')) -and (-not(RecipientTypeDetailsValue -eq 'DiscoveryMailbox')) -and (-not(RecipientTypeDetailsValue -eq 'ArbitrationMailbox')))
Note that everything after the two recipient types is added automatically when you set the filter. So, don't include that information when you set a filter. If you do, those requirements will be in there twice.

Unfortunately, the recipient type UserMailbox includes on-premises resource mailboxes and the MailUser recipient type includes cloud resource mailboxes. To exclude those from our dynamic distribution group we need to manually configure the filter. We can do this by excluding the following values for RecipientTypeDetails:
  • RoomMailbox
  • EquipmentMailbox
  • RemoteRoomMailbox
  • RemoteEquipmentMailbox
I set the following filter to achieve this:
((((RecipientType -eq 'UserMailbox') -or (RecipientType -eq 'MailUser'))) -and (-not(RecipientTypeDetailsValue -eq 'RemoteRoomMailbox')) -and (-not(RecipientTypeDetailsValue -eq 'RoomMailbox')) -and (-not(RecipientTypeDetailsValue -eq 'RemoteEquipmentMailbox')) -and (-not(RecipientTypeDetailsValue -eq 'EquipmentMailbox')))
Now, the next challenge is allowing cloud users to see this dynamic distribution group. This is an issue because Azure AD Connect does not synchronize dynamic distribution groups to Office 365. For more information about this see:



Wednesday, May 10, 2017

Removing a Cloud Mailbox from A Synced User

The title for this post is a bit incomplete because there wasn't enough space for a full explanation. Here is the scenario I had...

Before setting up hybrid mode for an Exchange 2010 organization, a user account was created in Office 365 with the same UPN as a user account in on-premises. The on-premises domain had been added to Office 365, but directory synchronization was not yet in place.  As part of testing, the user account had been give an Office 365 license, which then created an online Exchange mailbox.

When directory synchronization was configured, the on-premises AD account was matched with the existing cloud user. However, the on-premises user has an on-premises mailbox and the cloud user has a cloud mailbox. When viewing the cloud user in Office 365, you could see the mailbox in the cloud.

Because there was an existing cloud mailbox I couldn't move the on-premises mailbox up to Office 365. There was also no way to remove the cloud mailbox to allow the on-premises mailbox to replace it. If the user license was removed, then the cloud mailbox was removed, but came back again when the license was re-added.

To fix this I had to delete the cloud user account and recreate it. To delete the cloud user account, I moved it into an organizational unit that Azure AD Connect was not syncing. Then I purged the user account in Office 365 with the following cmdlet:
Remove-MsolUser -UserPrincipalName user@domain.com -RemoveFromRecycleBin
Finally I moved the user back to an OU that was synchronized. This recreated the cloud user account without a mailbox and the account was identified as having an on-premises mailbox.

Monday, May 8, 2017

New Hybrid Free/Busy Fails Cloud to On-Prem

When you configure hybrid mode, calendar sharing is automatically configured between on-premises and cloud users. You don't need to configure anything. However, in some cases, you may need to perform an IISreset.

On a recent project implementing implementing hybrid mode for an Exchange 2010 organization, we enabled hybrid mode and tested connectivity. Everything looked good for mail flow and mailbox moves. However, free/busy lookups failed from the cloud users to on-premises. When we tried to view the on-premises users' calendars the status was "No Information".

When you're not sure what's going on, it's always a good idea to use the Remote Connectivity Analyzer to test things out. There is an Office 365 tab which includes a Free/Busy test. When I ran this test, it was successful, as shown in the figure below. However it still wasn't working.

It's been a while since I've implemented Hybrid for Exchange 2010. With a quick search, I found this blog post that reminded me that an IIS reset will fix this up:
Most Exchange IIS related issues resolve themselves after a period of time. That period of time might be 12-16 hours, but usually there is some background process that refreshes things.

In this case, hybrid mode had been configured for multiple days. The free/busy lookups required an IISreset or a server reboot to fix.

This fix is certainly relevant for Exchange Server 2010. It may also be relevant for later versions of Exchange Server.

Wednesday, April 26, 2017

Must Run O365 Hybrid Wizard with IE

Today I made the mistake of downloading and attempting to run the Office 365 Hybrid Deployment Wizard by using Firefox. When you do, the app starts to launch and the fails. When you click on the Details button to open the log file, you see this under the error summary:
Deployment and application do not have matching security zones.
Download the app from within IE and no problems.

Link to download the wizard:
UPDATE: Also worth noting that a similar error with the wizard not running occurs if the .application file extension is not associated with Internet Explorer.

Tuesday, April 18, 2017

Script to Synchronize Primary Email Address with UPN

When planning an Office 365 implementation, it is best practice to start by assuming that UPN for signing in to Office 365 should match the user email address. If you don't configure it this way, then users have two separate items (their UPN for signing in and their email address) that look very similar. In many cases users are confused by the similarity.

If you are synchronizing  your on-premises Active Directory with Office 365 (in most cases you do) then you need to set the UPN for the on-premises user accounts with the correct UPN. The UPN from on-premises user accounts is synchronized to Office 365 to create the ID for signing in.

Most organizations are not using the UPN on user accounts for authentication on-premises. The option has been there since Windows 2000, but most organizations still use the domainname\username format for authentication. However, you need to verify if any user accounts are using the UPN for authentication before making this change. At minimum, you should communicate with your application and system administrators to see if they are aware of anything that might use UPNs. If your organization has issued certificates to users, they might be using UPN as the unique identifier for the certificate.

The script below does the following:
  • Obtains a list of all users where the proxyAddresses attribute has a value. This is done so that the result include only user accounts with an Exchange attributes configured.
  • Identifies the primary email address based on the all caps "SMTP:" text.
  • Strips out the "SMTP:" text from the primary SMTP address.
  • If the new UPN and the existing UPN do not match the user account is updated and the change is logged.
The location and name of the log file are configured in $logfile. You need to manually configure this variable and verify that the necessary folders exist.

  
 #Log folder must already exist  
 $logfile = "C:\Scripts\SyncUPN.txt"  
   
 #Adds timestamp to log file  
 Get-Date | Out-File -FilePath $logfile -Append  
   
 #Obtains only users with valid proxyAddresses attribute   
 $users = Get-ADUser -Properties proxyAddresses -Filter {proxyAddresses -like "*"}  
   
 #Prepare variables for processing status  
 $total = $users.count  
 $current = 0  
   
 Foreach ($u in $users) {  
   #Find primary SMTP address for user  
   $primarySMTP = $u.proxyAddresses | Where-Object {$_ -clike "SMTP:*"}  
       
   #Remove "SMTP:" to create the new UPN value  
   $newUPN = $primarySMTP.Substring(5)  
       
   #Set the new UPN value only if required  
   If ($u.UserPrincipalName -ne $newUPN) {  
     $u.DistinguishedName + " Old UPN: " + $u.UserPrincipalName | Out-File -FilePath $logfile -Append  
     $u.DistinguishedName + " New UPN: " + $newUPN | Out-File -FilePath $logfile -Append  
     Set-ADUser $u -UserPrincipalName $newUPN      
   } #end if  
   
   #Processing status  
   $current += 1  
   Write-Progress -Activity "Processing users to update UPN to primary email address" -Status "Progress: $current" -PercentComplete ($current/$total*100)   
        
 } #end foreach  

Script to Remove Old Domains from User Email Addresses

When managing email addresses and domains in Exchange Server, old email addresses are never removed automatically. This is good because it ensures that email addresses on a mailbox are never accidentally lost. However, you may want to clean up old domains or address formats that are no longer in use.

Some common scenarios where you might want to remove an old domain:
  • An SMB deployment of Exchange Server where a .local domain was added as the first domain for email addresses.
  • Old GroupWise addresses are left in place from an older migration.
  • Obsolete domain left over from a company merger many years ago
I often find that obsolete domains are identified when I run IDFix as part of preparing to migrate to Office 365. To simplify the removal of obsolete domains, I have created the following script.

A few things to note:
  • You need to set $RemovePattern to identify the domain to be removed. Any email addresses matching this pattern will be removed from proxyAddresses attribute in Active Directory objects.
  • The script uses Get-ADObject rather than Get-ADUser to make sure that the domain is removed from distribution groups too.
  • This version of the script is capable of removing multiple instances of a matching email address. So, if a user has several email addresses in the old domain, all of them are removed.
  • At the end of the script, I use Write-Progress to display a status bar. It's not necessary, but if there is a large number of users it's nice to see activity on the screen instead of just waiting and hoping it's doing something.

 #This pattern is used to match the email addresses being removed.  
 #Test that this pattern finds the correct users and email addresses  
 #before running this script.  
 #Example: $pattern = "smtp:*@olddomain.com"  
 $RemovePattern = "smtp:*@olddomain.local"  
   
 Import-Module ActiveDirectory #only required for 2008 R2  
   
 #Get the users that have an email address that matches the pattern  
 Clear-Host  
 Write-Host "Querying objects...This may take a moment"  
 Write-Host ""  
 $objects = Get-ADObject -Filter {ProxyAddresses -like $RemovePattern} -Properties ProxyAddresses  
   
 #Identify address being removed from first user for warning  
 [String]$proxyexample = $objects[0].proxyaddresses -like $RemovePattern  
   
 #Display warning and get confirmation  
 Write-Host "You are going to remove email addresses that match the following pattern:"  
 Write-Host -ForegroundColor Red "$RemovePattern"  
 Write-Host ""  
 Write-Host "This is an example from the first object:"  
 Write-Host -ForegroundColor Red "$proxyexample"  
 Write-Host ""  
 Write-Host "This will modify $($objects.count) objects"  
 Write-Host ""  
   
 $confirm = Read-Host "Enter Y to continue"  
 If ($confirm -ne "Y") {Break}  
   
 #Prepare variables for processing status  
 $total = $objects.count  
 $current = 0  
   
 #Processing users to remove addresses  
 Foreach ($o in $objects) {  
     
   #Build list of addresses to remove for object  
   #required because there might be multiple that match  
   $proxy= New-Object System.Collections.ArrayList  
   
   Foreach ($a in ($o.ProxyAddresses)) {  
     If ($a -like $RemovePattern) {  
       $proxy.add($a) | Out-Null  
     } #end if  
   } #end foreach  
     
   #Remove each bad address  
   Foreach ($p in $proxy) {  
     Set-ADObject $o -Remove @{'proxyAddresses'=$p}  
   }  
     
   #Pause  
   
   #Processing status  
   $current += 1  
   Write-Progress -Activity "Removing email addresses that match pattern" -Status "Progress: $current" -PercentComplete ($current/$total*100)   
 } #end foreach  

Thursday, April 13, 2017

Change All UPNs in a Domain

I needed to update all UPNs in a domain today. It was pretty quick to figure out, but here is one line to take care of it for you.

Get-ADUser -Filter * | ForEach-Object { Set-ADUser $_ -UserPrincipalName ($_.UserPrincipalName).Replace("OldDomain","NewDomain")}
Remember to make the pattern in the OldDomain unique enough that you don't accidentally change things you don't intend to. For example, if you are changing from a .local domain in the UPN to a .com, make sure that you replace ".local" and not "local" on the off chance one of the user IDs includes "local" in the name.

If there are any user accounts without a UPN, then an error is generated for those accounts. My domain had 4 accounts without a UPN:
  • krbtgt - default account used for kerberos
  • IWAM_ServerName - Old IIS account from Windows 2003
  • IUSR_ServerName - Old IIS account from Windows 2003
  • support_XXXXXXX - Used by Help and Support service

Suppress Results when Adding Items to an ArrayList

I ran into a mildly annoying feature when adding items to an array list when using PowerShell today. An array list is an expandable array of items with better performance than a normal array when working with large data sets.

Each time I added an item to the array list, it echoed back the index number of the list item. When I added the first item in the list, the number 0 was displayed on the screen. Adding a second item would echo back the number 1. For example:
$proxylist.add($a)
0

I would prefer my script to be silent when running except when there is data that I want to display. However, there is no option obviously available for that purpose. Instead, you need to redirect the output to $null. There are a few ways to do this and any one will work:
$proxylist.add($a) > $null
$proxylist.add($a) | Out-Null
[void]$proxylist.add($a)

Sunday, April 2, 2017

Dell Open Manage System Administrator Hangs (or Unavailable)

Just ran into an issue on Dell servers using the Dell Open Manage System Administrator software. This software runs on the server to let you see hardware details such as failed components and RAID configuration.

My first issue was when running the System Administrator icon from the desktop. This icon opens up and web page to access System Administrator. However, when Internet Explorer was launched, it came up with the error:
This page can't be displayed
So, I did the standard stuff:
  • restart services
  • verify DNS resolution
  • verify port 1311 is not blocked by firewalls and is listening
Everything looked good, but it wasn't working. One person on a discussion group indicated that they found it was because the older versions of System Administrator used older encryption algorithms for TLS and so the browser was blocking connectivity.

I attempted to resolve it first by updating the existing installation of Server Administrator. This changed the problem to hanging while trying to access the app, but didn't fix it.

The final fix was to remove older versions of System Administrator and install the latest version fresh. It seems that upgrading kept some older incorrect settings. The new install wiped out the older settings and all was good.

So, if Server Administrator is reporting "This page can't be displayed" or hanging when you attempt to access it, try an uninstall and reinstall. You don't need to reboot.