Showing posts with label Exchange 2013. Show all posts
Showing posts with label Exchange 2013. Show all posts

Wednesday, May 30, 2018

Expired Microsoft Exchange Server Auth Certificate

When you install your first Exchange Server 2013 or Exchange Server 2016 server, a certificate with the friendly name Microsoft Exchange Server Auth Certificate is created. This certificate is self-signed and used for OAuth authentication between applications such as Exchange Server and SharePoint. However, it is also used for hybrid deployments between on-premises Exchange Server and Exchange Online.

This certificate is unique because it is installed on all of your Exchange servers. The subject for the certificate is "CN=Microsoft Exchange Server Auth Certificate" and does not contain any SAN names with references to specific servers.

It also has a 5-year lifetime. Which is just long enough for everyone to forget about it. I suspect that this certificate is due to expire in many organizations soon.


Today I got a call from an organization with the following symptoms:
  • Outlook clients were slow to start
  • Outlook clients were not displaying the user's calendar
  • Outlook clients were not able to access Out-of-Office settings
  • Everything was functioning fine the previous day
  • No system changes had been made
  • The organization is configured in hybrid mode with Exchange Online, but the issue was occurring for on-premises users
  • Accessing through OWA was unaffected
An initial review of the configuration didn't identify anything out of the ordinary except that the Microsoft Exchange Server Auth Certificate had expired about two weeks ago. There was also this related event in the Application log:


The event information is:

  Source: MSExchange Web Services
  Event ID: 24
  
  The Exchange certificate [Subject]
  CN=Microsoft Exchange Server Auth Certificate

  [Issuer]
  CN=Microsoft Exchange Server Auth Certificate

  [Serial Number]
  3329C02443B435A447B92E81EB177303

  [Not Before]
  2013-06-11 6:25:31 PM

  [Not After]
  2018-05-16 6:25:31 PM

  [Thumbprint]
  DFC06BF3FC76974BCD1C050340998B65D5491007
  expired on 2018-05-16 6:25:31 PM.

This certificate is not bound to IIS, so why is this expiration being noted by MSExchange Web Services?

The certificate used for authorization is set separately from IIS but is configurable. You can see the currently configured certificate by using Get-AuthConfig. The thumbprint for the certificate is listed in the CurrentCertificateThumbprint property.


If you've already been searching around related to this certificate, you've probably found some instructions on how to recreate it. However, if it is expired, you can just renew it instead by using the Exchange Admin Console. In servers > certificates, select Microsoft Exchange Server Auth Certificate and then click Renew in the details pane as shown below. The screen shot below is of a certificate that is not expired yet, it looks exactly the same other than the expiry date.

Renewing creates a second certificate named Microsoft Exchange Server Auth Certificate that is valid for another 5 years. This certificate has a new thumbprint and exists only on the server you've renewed it on. You need to identify the thumbprint for the new certificate. If you edit the certificate, in Exchange Admin Center, the thumbprint is on the general tab as shown below. You can type this in, but you're probably better of to cut and paste it into the later commands.





My preference is to put the required information for the next command into variable. We need the thumbprint of the new certificate and the current date.

$thumb = "NewCertificateThumbprint"
$date = get-date

Then run the following command to add the new certificate:

Set-AuthConfig -NewCertificateThumbprint $thumb -NewCertificateEffectiveDate $date

You will get a warning that the new effective date is not 48 hours in the future. However, if we're recovering from an expired certificate, we're OK with that.

Now you need to publish the certificate to all servers:

Set-AuthConfig -PublishCertificate

And finally, remove the old expired certificate from the configuration:

Set-AuthConfig -ClearPreviousCertificate


Finally, since we've removed all references to the expired certificate you can delete it from all Exchange servers.

You may find that you need to do an iisreset after all of the AuthConfig changes were done. I found some examples where they ran iisreset.exe and didn't take the time to experiment. It's possible that it will pickup without a restart but just take longer.

I've added another post related to the Auth certificate and hybrid deployments here:




Wednesday, September 27, 2017

Automating Let's Encrypt DNS Verification with GoDaddy DNS for Exchange

The script that I reference in this post can be downloaded here: GoDaddyDNSUpdatePublic.ps1.txt

I love the concept of using Let's Encrypt for free SSL/TLS certificates. However, the short 90-day lifetime of the certificates is designed for automated renewal. In this blog post I'm going to show the steps required to script the use of GoDaddy for DNS verification.

For the basic steps on how to get a SAN certificate by using Let's Encrypt and DNS verification by using Windows PowerShell, please see my previous blog post: Using Let's Encrypt Certificates for Exchange Server

Let's Encrypt requires you to create an identifier for each DNS name that you want to include on a certificate. You need to validate each identifier to prove ownership of the domain. When you are using DNS validation, you need to create a TXT record in DNS for each identifier.

Unfortunately (from an ease of user perspective), the validation for an identifier is only valid for 30 days. This means, when you go to renew a certificate, you also need validate your identifiers again. Practically, this means you create new identifiers with the same DNS name, but a different alias, and validate them before generating the certificate.

Since DNS validation requires you to create a TXT record in DNS, you need a way to automate this. Fortunately, many DNS providers have a web API that you can use to programmatically access and create DNS records. However, be aware that there is no wide spread standard for this API. GoDaddy has created DomainConnect and submitted it as a standard, but I've not seen wide acceptance of it.

For this blog, I'll be showing the use of GoDaddy's API mostly because it's the DNS provider that I use most often.

Authentication to create and manage DNS is done by using an API key and a secret. Both of these are included in the URL when you perform tasks. You get the API key and secret from the GoDaddy Developer Portal (developer.godaddy.com).

On the GoDaddy Developer portal:
  1. Sign-in by using your GoDaddy Account
  2. Go to the Keys tab and click Create Key.
  3. Give your key a name.
  4. Copy the key and the secret to a file and click OK.
This creates a test key which you cannot use for updating your DNS. However, you're now at a screen where you can create a production key. Save the details of that production key to a file for using in the script.

At this point, it is assumed that you've already created a vault and registered by using the ACMESharp cmdlets. The remaining steps are purely to automate the process.

 #define domain that records are being created in  
 #script only supports a single domain  
 $domain = 'yourdomain.com'  
   
 #For Pfx file  
 $pfxPass = "PasswordOfYourChoiceToSecurePfxFile"  
 $pfxPath = "C:\Scripts\cert.pfx"  
   
 #header used for accessing GoDaddy API  
 $key = 'YourBigLongKeyHere'  
 $secret = 'YourBigLongPasswordHere'  
 $headers = @{}  
 $headers["Authorization"] = 'sso-key ' + $key + ':' + $secret  
   
 #First identity will be the subject, all others in SAN   
 $identities = "mail.yourdomain.com","autodiscover.yourdomain.com" 

 $allAlias = @()

I started my script by defining some variables used later on:
  • $domain is your domain in GoDaddy where the DNS records are being created.
  • $pfxPass and $pfxPath are used used then the certificate is exported to a PFX file before being imported into the Exchange Server.
  • $key and $secret are provided by GoDaddy when you obtain your production key for the API.
  • $headers is included as the authentication information later on when the call is made to the GoDaddy web api to create the TXT record.
  • $identities contains the DNS names that will be included in the certificate. My example has two names, but more names can be added.
  • $allAlias is defined as an array so that later functionality adding aliases can be processed.

 Foreach ($ident in $identities) {  
   [string]$unique = (Get-Date).Ticks  
   $alias = ($ident.Replace(".",""))+$unique  
   $allAlias = $allAlias + $alias  
   $id=New-ACMEIdentifier -Dns $ident -alias $alias   
   If ($id.Status -eq "valid") {Continue}  
   $chResults = Complete-ACMEChallenge $alias -ChallengeType dns-01 -Handler manual  
   $chData = $chResults.Challenges | Where-Object {$_.Type -eq "dns-01"}  
   $value=$chData.Challenge.RecordValue  
   $name=$chData.Challenge.RecordName  
     
   #remove domain name from $name  
   $recordname = $name.Replace(".$domain","")  

I use a Foreach look to create each identity and verify it using DNS. I'm going to go through this Foreach loop in chunks.

Since I'm going to need to create multiple identities over time, I wanted a unique identifier ($unique) to ensure there wouldn't be conflicts in naming. I chose to use the ticks value from time. This has the added advantage that you could sort them based on when they were created.

Each identity is referred by by it's alias. I defined the alias for each identity as the DNS name of the identity with the dots removed and $unique added. After each alias is generated, it's added to the $allAlias array.

When the identifier is created for Let's Encrypt, it's placed in the $id variable. The $id variable is then used to verify the status of the identifier. If an identifier with the same DNS name has already been created and verified then the status is valid. If it's valid, we don't need to do any of the other work in the loop and Continue to tell the script to carry on with the next identifier. If the status is not valid (which is expected for new identifiers) then we process the rest of the loop.

The results for the Complete-ACME challenge are placed in $chResults. The Challenges property for those results for the dns-01 challenge type are placed in $chData where we can get the RecordValue and RecordName properties:
  • $name contains the name of the TXT record required for validation
  • $value contains the text string that needs to be included in the TXT record for validation
When the TXT record is created by using the GoDaddy API, the data used in the request does not contain the domain name in the name. The $name variable is processed to remove the domain name (the domain name is replaced with nothing) and the results placed in $recordname which contains the data that will be submitted to the GoDaddy API.

 #create DNS record  
 $json = ConvertTo-Json @{data=$value}
 Invoke-WebRequest https://api.godaddy.com/v1/domains/$domain/records/TXT/$recordname -method put -headers $headers -Body $json -ContentType "application/json"  

UPDATE: July 2, 2018
A reader named Jason has reported that the json format used by GoDaddy has changed and that the above code snippet needs to be updated. I have not verified, but Jason says updating the middle line to the following fixes the problem:
  • $json = ConvertTo-Json @(@{data=$value})


After all the processing of data is done, creating the TXT record is fairly straightforward. The data for the request is put into a hash table that is converted into Json. This hash table only requires the data but you can include other information like the TTL.

Invoke-WebRequest accesses the GoDaddy web api with a put method to send the data. This same method that web forms use to return data to a web site. The URL being access needs to contain your domain name and the type of record being created. In this case, I hard coded TXT as the record type in the URL, but $domain is used to insert the domain name. The $recordname variable is included in the URL because we only want to create that specific record. If the URL ends at TXT then the API assumes that we're providing an array of all the TXT records and any other existing TXT records are wiped out. The $headers variable (defined earlier) provides the authentication information for the request.

 #Submit the challenge to verify the DNS record  
 #30 second wait is to ensure that DNS record is available via query  
 Do {    
   Write-Host "Waiting 30 seconds before testing DNS record"  
   Start-Sleep -Seconds 30
   $dnslookup = Resolve-DnsName -Type TXT -Name $name  
   $dnsSuccess = $false  
   If ($dnslookup.strings -eq $value) {  
     Write-Host "DNS query was successful"  
     Submit-ACMEChallenge $alias -ChallengeType dns-01  
     $dnsSuccess = $true  
   } Else {  
     Write-Host "DNS query for $name failed"  
     Write-Host "The value is not $value"  
     Write-Host "We will try again"  
   }  
 } Until ($dnsSuccess -eq $true)   
   

I ran into an issue when verifying the TXT records. After creating the TXT record there can be a delay (a few second to a few minutes) until the record is accessible via the DNS servers. If Let's Encrypt tries to verify before it accessible, then it fails and isn't recoverable. You need to create a new identifier. So, I added this code to verify the DNS record is accessible before telling Let's Encrypt to verify.

The Resolve-DnsName cmdlet looks for the TXT record we just created. If the lookup contains the expected value then Submit-ACMEChallenge is used to tell Let's Encrypt to verify it. Also, the variable $dnsSuccess is set to $true and the loop ends. If it's not successful, we try again at 30 second intervals until it resolves successfully. Since adding this code to the script I haven't had any failures from Let's Encrypt. I think there may be some caching of failed lookups on the client running the script which result in a two minute delay, but better that than the Let's Encrypt lookup failing.

 #Verify that the dns record was checked and valid  
 $chStatus = $null  
 Do {  
   Write-Host "Waiting 10 seconds before testing DNS validation"  
   Start-Sleep -Seconds 10    
   $chStatus = (Update-ACMEIdentifier $alias).Status   
   #$chStatus=((Update-ACMEIdentifier $alias -ChallengeType dns-01).Challenges | Where-Object {$_.Type -eq "dns-01"}).Status  
   If ($chStatus -eq "valid") {Write-Host "$ident is verfied"}  
   Else {Write-Host "DNS record not valid yet: $chStatus"}  
 } Until ($chStatus -eq "valid")  
   

After the DNS lookup for the TXT record is successful, the script uses Update-ACME to retrieve the status of the verification. There is a 10 second pause to allow Let's Encrypt to perform the verification. If the verification is still pending then the loop repeats again at 10 second intervals.

I have two methods for checking the status ($chStatus). The more complex version is one I saw in an example someone else provided. However, I found that the simpler version seems to work fine. However, I did see one person indicating that when the challenge type is not specified that a pending request is not properly retained in the local vault and fails. With my delay of 10 seconds, I'm not sure that's ever happened. Both versions do seem to work though.

This is the end of the Foreach loop that processes the identifiers. Each identifier has now been verfied and there is a variable $allAlias that contains the alias used for each identifier. Next up is creating the certificate.

 #Create Certificate  
 New-ACMECertificate $allAlias[0] -generate -AlternativeIdentifierRefs $allAlias -Alias $allAlias[0]  
 Submit-ACMECertificate $allAlias[0]  
 Write-Host "Waiting 10 seconds for certificate to be generated"  
 Start-Sleep -Seconds 10  
 Update-ACMECertificate $allAlias[0]  
 Get-ACMECertificate $allAlias[0] -ExportPkcs12 $pfxPath -CertificatePassword $pfxPass -Overwrite  
   

The New-ACMECertificate cmdlet creates a certificate request. The first identifier ($allAlias[0]) becomes the subject of the certificate. Then the entire array $allAlias is submitted as alternative references which is the Subject Alternative Names attribute in the certificate. When you list the alternative identifiers manually, you can skip the identifier used for the subject because it's automatically added to the SAN attribute also. However, it works fine when that identifier is specified also. The alias for the certificate is set to be the same as the alias for the identifier used as the subject.

The certificate request is submitted and the script waits for 10 seconds to ensure that Let's Encrypt has time to generate the certificate. Update-ACMECertificate retrieve the certificate information from Let's Encrypt and puts it in the local vault.

Get-ACMECertificate with the -ExportPkcs parameter is used to export the certificate to a PFX file that can be imported into Exchange Server. While a password is not required while overwriting, the import of a PFX file without a password won't work properly. All will appear good, but the certificate will behave as if it has no private key. The -Overwrite parameter is specified because it's assumed that this script will be automated and this allows the file generated to be overwritten each time.

 #Assign certificate using EMS commands  
 #If run as a scheduled task must be run at EMS prompt  
 $pfxSecurePass = ConvertTo-SecureString -String $pfxPass -AsPlainText -Force  
 $cert = Import-ExchangeCertificate -FileName $pfxPath -Password $pfxSecurePass -FriendlyName $allAlias[0] -PrivateKeyExportable $true  
 Enable-ExchangeCertificate $cert.Thumbprint -Services IIS,SMTP -Force  
   

Finally, we import the certificate into Exchange Server. The Import-ExchangeCertificate cmdlet won't accept a plain text password. So, the password is converted to a secure string that can be used and placed into $pfxSecurePass.

To make it easier to identify the certificate on the Exchange server, the alias is used as the friendly name. The private key is also marked as exportable because by default it is not.

After the certificate is imported, then Enable-ExchangeCertificate is used to specify that it should be applied to the IIS and SMTP services. The -Force parameter enables the cmdlet to complete without requiring user input. However, it will replace the default SMTP certificate with this option.

Because the Exchange cmdlets are used, you need to schedule this script so that it runs in the Exchange Management Shell and not a regular PowerShell prompt.

If you are going to use this certificate on multiple Exchange servers, then you should update this script to import and enable the certificate on all of the Exchange servers in the site. The current configuration assumes one Exchange server and it applies only on the local Exchange server where the script is run.

Is it worth it?

In reality, the complexity of keeping this up an running is probably not worth it for Exchange Server. Exchange Server is a mission critical part of your organization. Instead I'd go for a low cost certificate provider and get a cert that lasts for 3 years. A SAN certificate from NameCheap.com costs about $35US per year. That's far less than the value of your time getting this up and going.

That said, this was a fun exercise for me and I'll probably use this for test environments. Now that I have the script it's pretty easy for me to use it. 

Alternative for IIS

As part of developing this, I worked out a method to supply the certificate only to IIS. I don't think that it's very useful for Exchange Server since we also want it to apply to SMTP to allow secured SMTP communication. However, I'm including it in case anyone is interested.

 #Import certificate from pfx to server  
 #script must be running as administrator to perform this action  
 $cert=Import-PfxCertificate -FilePath $pfxPath -CertStoreLocation Cert:\LocalMachine\My -Password $pfxSecurePass -Exportable  
   
 #Assign certificate to default web site  
 Import-Module WebAdministration #required for IIS: PSDrive  
 Set-Location IIS:\SSLBindings  
 Get-ChildItem | Where-Object { $_.sites.Value -eq "Default Web Site" -and $_.Port -eq 443 -and $_.IPAddress.IPAddressToString -eq "0.0.0.0" } | Remove-Item  
 $cert | New-Item .\0.0.0.0:443  
   

Importing the script is done with the Import-PfxCertificate. Again, we specify a path, password, and mark it as exportable.

We manually import the WebAdministration module to get access to the IIS: PSDrive. In Windows Server 2012 and later, modules autoload for using cmdlets, but not PSDrives.

SSL bindings are located in IIS:\SSLBindings. So, the location is set to there. In that location, Get-ChildItem gets a list of the SSL bindings. The binding for all IP addresses on port 443 of Default Web Site is deleted.

The certificate information is piped to New-Item with .\0.0.0.0:443. This creates a new binding in the current directory for that certificate to 0.0.0.0 on port 443.

References:

Thursday, September 14, 2017

Getting Detailed Error Messages for Mailbox Moves

In Office 365 or Exchange Server 2013/2016, you can use the administration console to view information about migration batches. To find out information about failing moves, you can view the details of the migration batch and then view the report for individual mailboxes. When you view the report for a mailbox a text file is downloaded for viewing.

The report provides detailed information about how much data has been downloaded. Also, if there are errors, they are contained in the report. Unfortunately sometimes the errors are pretty generic. For example, one error I got recently was:
Transient error TimeoutErrorTransientException has occurred. The system will retry (200/1300).
Instructions on how to review the report:
Since the error was happening often, we needed to get more information. Fortunately that detail is available, but not in that report. Instead, you need to use Windows PowerShell to view the move request statistics. If you are moving the mailbox to Office 365, you need to use a PowerShell prompt connected to Exchange online.

$stats = Get-MoveRequest user@domain.com | Get-MoveRequestStatistics -IncludeReport
$moveErr = $stats.Report.Entries | Where-Object {$_.Type -eq "Error"}

This leaves you will an array named $moveErr that contains all of the errors. You can view each error by specifying an individual array item.

$moveErr[1]

In my case, I got this more detailed information in the Failure property:

TimeoutErrorTransientException: The call to 'https://hybridserver/EWS/mrsproxy.svc
hybridserver (15.1.1034.26 caps:07FD6FFFBF5FFFFFCB07FFFF)' timed out. Error
details: The request channel timed out while waiting for a reply after 00:00:30.9527011.
Increase the timeout value passed to the call to Request or increase the SendTimeout
value on the Binding. The time allotted to this operation may have been a portion of a
longer timeout. --> The request operation did not complete within the allotted timeout of
00:00:50. The time allotted to this operation may have been a portion of a longer
timeout. --> The request channel timed out while waiting for a reply after
00:00:30.9527011. Increase the timeout value passed to the call to Request or increase
the SendTimeout value on the Binding. The time allotted to this operation may have been a
portion of a longer timeout. --> The request operation did not complete within the
allotted timeout of 00:00:50. The time allotted to this operation may have been a portion
of a longer timeout.

That error information gave me enough information to start looking at timeout settings for the MRS proxy service.

Tuesday, September 12, 2017

Using Let's Encrypt Certificates for Exchange Server

Have you ever fantasized about using free SSL/TLS certificates for Exchange Server? If so, then this blog post is for you.

I’ve always hated the cost associated with SSL/TLS certificates. For what seemed like a pretty basic service some of the certificate authorities (CAs) were charging hundreds or thousands of dollars. You could always set up your own CA, but that didn’t work well with random clients on the Internet because they won’t trust certificates generated by your CA.

At the end of 2015, there was a game changing development. Let’s Encrypt started giving away SSL/TLS certificates for free. At the time, the certificates were only for a single name. So, without SAN support, not good for Exchange Server. However, now there is support for SAN/UCC certificates. And, in 2018 they are planning to support wildcard certificates.

What’s the Catch?

The certificates are free. There is no catch there. But, they do have a short lifetime of 90 days. The short lifetime is to ensure that compromised certificates are not available for an extended period of time. Because of the short lifetime, it is strongly recommended that you automate certificate renewal.

Note: This blog post only shows the manual steps for obtaining a certificate. I'll put up another one showing automation.

The process for generating and renewing a certificate is a bit complex. But, once the initial process is defined, it’s pretty easy to work with.

Unlike a typical CA, Let’s Encrypt does not provide a web site to manage your certificate requests. Instead you need client software that communicates with the Let’s Encrypt servers. Since I already work with Windows PowerShell on a regular basis, I like the ACMESharp module that provides PowerShell cmdlets for working with Let’s Encrypt.

Installing the ACMESharp Module

The ACMESharp module is available in the PowerShell Gallery. To download and install modules from the PowerShell Gallery, you use the Install-Module cmdlet that is part of the PowerShellGet module. The PowerShellGet modules is included as part of the Windows Management Framework 5 (part of Windows 10 and Windows Server 2016).

If you are not using Windows 10 or Windows Server 2016, you can download and install WMF 5 or a standalone MSI installer for PowerShellGet here:

After you have PowerShellGet, installed run the command:
Install-Module AcmeSharp

When you run this command, you might be prompted to install NuGet. If you are prompted, say yes to install it. NuGet is provides the functionality to obtain packages from the PowerShell Gallery. The PowerShellGet cmdlets use NuGet.

You might also be prompted that the repository PSGallery is untrusted. This is the PowerShell Gallery that you want to download files from. So, say yes to trust PSGallery.



Connecting to Let’s Encrypt


The first step after installing the ACMESharp module is creating a local data store for the ACMESharp client. The data store is used by the client for secure storage of requests and keys.
To create the local data store run the following command:
Initialize-ACMEVault

Then, to create an account with Let’s Encrypt, run the following command:
New-ACMERegistration -Contacts mailto:youremail@yourdomain.com -AcceptTos

Validating DNS Names

Let’s Encrypt requires you to verify ownership of each DNS name that you want to include in a certificate. Each DNS name is referred to as an identifier. For a SAN certificate, you will generate 2 or more identifiers then specify the identifiers when you create the certificate.

You can validate an identifier in three ways:
  • Dns - You need to create a TXT record in DNS that is verified by Let’s Encrypt.
  • HTTP - You need to place a file on your web server that is verified by Let’s Encrypt.
  • TLS-SNI - You need to place an SSL/TLS certificate on your web server that is verified by Let’s Encrypt.
In the projects I work on, I typically do not have access to the main company web server/site, but do have access to create DNS records. So, I use DNS validation.

To create a new identifier:
New-ACMEIdentifier -dns server.domain.com -alias idAlias
You should include an alias each time you create an identifier. If you don’t create an alias, there is no easy way to refer to the identifier in later steps. If you forget to create an alias, just create a another new identifier with the same DNS name and include the alias.

In my example, the DNS name has four parts because I was testing by using a subdomain. In most cases, the DNS name will have only three parts.


Next you need to specify how the identifier will be verified. When you do this the cmdlet reports back the proof you need to provide. For HTTP verification, it identifies the file name. For DNS verification, it identifies the TXT record that needs to be created.

The command to start verifying the identifier is:
Complete-ACMEChallenge idAlias -ChallengeType dns-01 -Handler manual
Use the alias of the identifier to specify which identifier is being verified.

The manual handler identifies that you are specifying the challenge type. There are other automated handlers that automatically create the response for the HTTP-based challenges. The automated handlers are specific to different web servers.

The challenge type dns-01 identifies that you will create a TXT record in DNS. Note that dns-01 must be in lowercase. 


After you have created the DNS record that corresponds to the challenge, then you submit the challenge. When you submit the challenge, Let’s Encrypt verifies it.

To submit a challenge:
Submit-ACMEChallenge idAlias -ChallengeType dns-01

When you submit the challenge, you need to specify the alias of the identifier and the challenge type.
The validation may or may not complete immediately. You can verify the status of the validation with the following command:
(Update-ACMEIdentifier idAlias -ChallengeType dns-01).challenges
The Update-ACMEIdentifier cmdlet queries the status of the identifier from the Let’s Encrypt servers. The challenges property contains the challenges generated when the identifier was created. The status for the dns-01 challenge will change to valid when validation is complete.



Creating a Certificate

After you have validated all of the identifiers that will be included in your certificate, you can generate the certificate request. When you generate the certificate request, you need to specify the identifiers to include and a new alias for the certificate.

To generate the certificate request:
New-ACMECertificate idAlias -generate -AlternativeIdentifierRefs idAlias1,idAlias2 -Alias certAlias
The initial alias identifier that you provide is the subject for the certificate. This name is also added to the subject alternative names automatically. You don’t need to repeat it as an alternative identifier.
The -AlternativeIdentifierRefs parameter is used to identify additional identifiers that are included in the certificate. All identifiers used here need to be validated.



To submit the certificate request:
Submit-ACMECertificate certAlias
The certificate alias used here is the alias that was set when running the New-ACMECertificate cmdlet.


After submitting, notice that the IssuerSerialNumber is blank. This is how you can identify that the results from the request you submitted have not been retrieved. You need to update the data in the local vault before you can export it:
Update-ACMECertificate certAlias

To export the completed certificate as a pfx file that includes the certificate and private key:
Get-ACMECertificate certAlias -ExportPkcs12 filename.pfx -CertificatePassword “password”
The -ExportPkcs12 parameter can be given a filename or a full path. If there are spaces in either one, you need to put quotes around it.


When I first ran the Get-ACMECertificate cmdlet to export the pfx file, I got an error:
Issuer certificate hasn’t been resolved.

While I found some references to this being an issue with an intermediate certificate not being installed, my issue was simpler. I had forgotten to update the certificate information in the local vault with Update-ACMECertificate before trying to export to a PFX file.

However, if you have this error and you have already used Update-ACMECertificate then it may be caused by the missing intermediate certificate. You want the X3 intermediate certificate for the Let's Encrypt CA.


If necessary, you can get the X3 intermediate certificate here:
After you have the pfx file, you can import it and assign to Exchange Server using the normal Exchange Management cmdlets.

Friday, June 23, 2017

Errors on Public Folder Migration

As I was doing a public folder migration today, I got a couple of errors that took me some time to resolve. These are caused by mail enabled public folders migrated from Exchange 2003. You will see these errors when you run Get-MailPublicFolder on Exchange 2010. Some of these errors will show up in the public folder migration logs when migrating to Exchange 2016. So, I prefer to clean these up first to ensure migration is successful.


Error #1

WARNING: The object domain.com/Microsoft Exchange System Objects/PF Name has been corrupted, and it's in an inconsistent state. The following validation errors happened:
WARNING: Could not convert property OnPremisesObjectGuid to type Guid. Byte array for GUID must be exactly 16 bytes long.
My best guess is that this property is left over from Exchange 2003 (or maybe earlier). The quick fix is to disable mail for the public folder and then mail-enable it again. However, when you do so, verify the email addresses before and after.

Error #2


WARNING: The object domain.com/Microsoft Exchange System Objects/PF Name has been corrupted, and it's in an inconsistent state. The following validation errors happened:
WARNING: Property expression "PF Name" isn't valid. Valid values are: Strings formed with characters from A to Z (uppercase or lowercase), digits from 0 to 9, !, #, $, %, &, ', *, +, -, /, =, ?, ^, _, `, {, |, } or ~. One or more periods may be embedded in an alias, but each period should be preceded and followed by at least one of the other characters. Unicode characters from U+00A1 to U+00FF are also valid in an alias, but they will be mapped to a best-fit US-ASCII string in the e-mail address, which is generated from such an alias.

This error is most commonly caused by a space in the Alias property. Update this property to remove spaces and the error should be gone.

Saturday, June 17, 2017

Multiple Moderation Approval Requests

I recently did a migration from Exchange 2010 to Exchange 2016 where the client uses a high volume of moderated messaging. There were over 100 transport rules that did message moderation of some sort. The initial deployment consisted of Exchange 2010 SP3 RU17 and Exchange 2016 CU4.

Deployment of Exchange 2016 into the Exchange 2010 environment didn't seem to have any effect. However, after we directed the internal namespace to Exchange 2016 for proxying, the approvals generated by the transport rules when whacky (yep that's the technical term).

Here is the process we saw:
  1. Message requiring moderation sent.
  2. Approval request sent to moderator.
  3. Moderator approves request
  4. Approval request sent to moderator
  5. Moderator approves request
  6. Repeat request and approval process a few more or a lot more times.
This process was happening even though we had not moved any mailboxes to Exchange 2016 yet. 

When searching, there were very few references to this issue on the Internet or support forums. However, there were a few suggestions that were consistent:
  • Ensure arbitration mailboxes are moved to Exchange 2016 (one of these stores the messages until they are approved).
  • Delete and recreate rules.
  • Move moderators mailbox to Exchange 2016.
  • Restart transport services.
All of these things were done but we still had issues. However, when both mailboxes were on Exchange 2016, the approvals on messages seemed to only happen twice. This was better than the random number from before.

I reviewed the message tracking logs for errors and didn't see any. In the logs, each time an approval was received, the message was released for delivery, but then promptly moderated again. However, the second attempt to approve worked.

All of my initial testing was done using inbound messages. So, I tried some scenarios with both mailboxes on Exchange 2016, and these were my results:
  • Inbound messages routing through Exchange 2010 first - Two approvals required
  • Outbound messages routing out through Exchange 2010 - One approval only
  • Inbound messages routing through Exchange 2016 only - One approval only
Based on these results, we can see that Exchange 2010 coexistence definitely plays a role, because when Exchange 2010 is not part of the inbound routing the issue doesn't occur. This at least provided confidence that after migration was complete the issue would not be persist.

The other item that needed to be addressed was the Exchange 2016 CU4. Microsoft releases updates in matched sets and Exchange 2016 CU4 was one step behind Exchange 2010 SP3 RU17. In the hope that having both at the same update level would fix it, we applied Exchange 2016 CU 5. We did the update to CU 5 but there was not change.

Final testing indicated that any message that entered the Exchange organization through Exchange 2010 was subject to double approval. This happened if the message came in from external or was generated by a mailbox in Exchange 2010. The location of the moderator mailbox did not make a difference.

So, to minimize the issue:
  • Move all inbound message routing to Exchange 2016 sooner rather than later. This includes Internet mail and applications that send messages to be moderated.
  • Move mailboxes that generate the most messages to be moderated first. Once the source is in Exchange 2016 the problem is mediated.
I should also note that there was a red herring in the application event log. We saw this error:
Event ID 1051, MSExchange Extensibility
Warning, MExRuntime
 Agent 'Approval Processing Agent' caused an unhandled exception 'SmtpResponseException: 250 2.1.5 APPROVAL.ApprovalRequestUpdated; approval request updated successfully' while handling event 'OnCreatedMessage'
However, review of the logs indicated that the error was present before the issue appeared. So it appears to be noise rather than useful information.

Tuesday, June 6, 2017

SourceMailboxAlreadyBeingMovedTransientException

Today while moving a mailbox from on-premises to Office 365 in a hybrid environment, I got the following error:
Transient error SourceMailboxAlreadyBeingMovedTransientException has occurred. The system will retry (5/620).
This error occurs when a previous move attempt did not get cleaned up properly. From a bit of reading, this should timeout and fix itself after about 2 hours. However, since I didn't want to wait that long, I did the following that got it going again.
  • IISReset.exe to restart the web services
  • Restart the Microsoft Exchange Mailbox Replication service
It is possible that only one of those two items was required, but I was more concerned about getting the move going than recording exact details.

Tuesday, October 25, 2016

Office 365 vs. On-Premises Exchange Server

A large client is currently running Exchange 2010 and is evaluating moving to Office 365 vs upgrading to Exchange Server 2016. I talked with them about it and thought it would be useful to document it for future reference.

If you are a very small organization, then Office 365 is a slam dunk. It's going to perform better and be more cost effective than your could ever implement on your own. This is even before we consider the cost of the the consultants to get your on-premises Exchange up and running.

For mid-sized and large businesses there are more things to think about....

Cost

Direct cost is the first thing everyone wants to evaluate when considering Office 365. Your exact costs are going to vary depending on how you want to implement Exchange and which Office 365 plans you think are appropriate. So, I'm going to let you figure out the exact costs, but here are the things you need to consider:

On premises:
  • Exchange Server licenses
  • Exchange Server CALs (basic and enterprise if required)
  • Hardware for the Exchange servers
  • Storage for the Exchange servers
  • Backup software and storage
  • Antispam and antivirus solution
 Office 365:
  • Monthly or yearly license (subscription-based)

High Availability

On-premises Exchange Server can be made highly available. This is done by configuring multiple Exchange servers in a database availability group (DAG) and by using load balancers. This configuration is well understood but definitely requires some planning and expertise to implement.  In most cases, hardware load balancers are used which are an additional cost.

Other high availability considerations for on-premises Exchange include redundant Internet connections, and alternate data centers. To provide a solution that is truly highly available, it takes a lot of redundant infrastructure.

Office 365 is highly available out of the box. You don't need to do anything, that's just the way it's designed. Your data is automatically located in multiple data centers in your region. If one data center fails, your data is still safe in another data center and you're still able to access it. This is one of the big advantages for Office 365, particularly for mid-sized organizations that might not have the resources to deploy office 365 across multiple data centers.

Authentication

On-premises Exchange Server uses Active Directory (AD) for authentication. When users are on a domain joined computer, authentication is performed automatically by Outlook. Externally, the AD username and password are used.

For Office 365, there are multiple ways that authentication can be configured. First lets talk about user accounts...

Very small organizations might choose to create user accounts in Office 365 manually. For these organizations, the users will sign in by using their email address and a password that is set in Office 365. This password could be different from the AD password on-premises and there is no synchronization of the passwords between AD and Office 365.

Larger organizations will use Azure AD Connect to synchronize user accounts with Office 365. This avoids the need to manually create the accounts. Azure AD Connect also has an option to synchronize passwords with Office 365. This simplifies password management for users. It also simplifies password resets when users call the help desk.

When passwords are synchronized with Office 365, all authentication is still handled by Office 365. There is no reliance on any on-premises infrastructure for the authentication process. So an on-premises service outage does not cause an Office 365 service outage.

Very large organizations may consider using Active Directory Federation Services (AD FS) for Office 365 authentication instead of password synchronization. When AD FS is implemented, Office 365 directs authentication requests to AD FS which then authenticates the request against AD. This is significantly more complex to configure than password synchronization but provides the following benefits:
  • True single sign-on where workstation credentials are passed through for authentication just like on-premises Exchange.
  • When an on-premises AD account is locked or disabled, this also applies to Office 365 authentication. Without this, disabling an Office 365 account is a separate process.
A key point to remember is that if your AD FS infrastructure is down, so is Office 365 authentication.

Account Lockout

For on-premises Exchange, the account lockout policies in AD are applied to authentication for Exchange mailboxes. The only exception is if a device such as a proxy is performing pre-authentication. If pre-authentication is performed then the device providing pre-authentication might have more restrictive policies than what is configured in AD.

When password synchronization is used with Office 365, there is a account lockout mechanism included in Office 365. After 10 failed attempts, an account is locked for 1 minute and if account lockout is triggered multiple times the lockout time is increased each time. This account lockout mechanism is not configurable.

When AD FS is used with Office 365, the account lockout policies in AD are used.

Mailbox and Archive Size

In on-premises Exchange, the maximum size of mailboxes and archive mailboxes is limited by the storage design. Or, more often by the cost of the storage being used. Using direct attached storage (DAS) with 7.2K SAS drives (rather than a SAN) is the most cost effective solution to provide large mailboxes and archives.

In Office 365, the maximum size of mailboxes and archive mailboxes is limited by the license assigned to the mailbox:
  • Kiosk: 2 GB mailbox, no archive
  • E1: 50 GB mailbox, 50 GB archive
  • E3: 50 GB mailbox, unlimited archive
  • E5: 50 GB mailbox, unlimited archive

Online Reading and Editing Attachments

In Exchange 2010, there was basic functionality for rendering attachments as web content in OWA. In Exchange 2016, this functionality is not included. Instead, you need to implement Office Online Server. Office Online Server is included with most volume licensing agreements for Exchange Server 2016.

Implementing Office Online Server is not terribly complex for a single server, but requires at least an additional virtual machine. If you want the functionality to be highly available you need to implement multiple servers and load balancing.

Office 365 includes the ability to read and edit attachments by default. No additional configuration is required.

Spam and Antivirus Filtering

For on-premises Exchange, you need to purchase additional products for spam and anti-virus filtering.

Office 365 licenses include Exchange Online Protection for anti-spam and virus protection at no additional cost.

Network Configuration

Most implementations of on-premises Exchange Server require you to implement proxies, load balancers, and firewalls. External access from the Internet to the Exchange servers also need to be configured.

For Office 365, you just need to ensure that clients on the internal network can communicate with Office 365 over the Internet.

Client Communication

Most on-premises deployments of Exchange server use Outlook in cached mode. However, because the internal network is reliable and fast, you can implement Outlook in online mode instead. This can be useful if you are concerned about the size of cached data on computers that are shared by multiple users.


Outlook clients must communicate with Office 365 servers over the Internet. This may result in higher Internet bandwidth usage. Generally, Outlook cached mode is used to minimize network utilization and provide better performance over networks with high latency.

Recent versions of Outlook cache only recent data in a mailbox rather than the entire mailbox. This mitigates concerns about large mailboxes being cached in local profiles. The amount of data cached is configurable.

Migration

Exchange Server 2016 can co-exist with Exchange Server 2010 or Exchange Server 2013. Migration is just a matter of moving mailboxes to Exchange Server 2016. There is almost no impact on users and the Outlook client is automatically redirected to the new mailbox location.

Office 365 can be integrated with Exchange Server 2010 or later by using hybrid mode. Once in hybrid mode, mailboxes can be migrated to Office 365 with minimal impact on users. Outlook is automatically reconfigured when mailboxes are moved.

Backup



Most common backup software can be used for backup with an appropriate agent on the Exchange servers. The restore functionality in the software varies but most can restore individual mailboxes or databases. This software is an extra cost.

There are third party options available for backing up Office 365 mailboxes, but there is no built-in functionality in Office 365 for this purpose.  However the same Single Item Recovery available in on-premises Exchange can be used in Office 365 to recover items for up to 30 days (default 14 days) after they are removed from deleted items.

Microsoft maintains multiple copies of mailbox data to avoid the need to restore from backup due to server error.

If you use E3 licenses, you can implement a hold on mailboxes to permanently retain messages received or sent from mailboxes. This held data is searchable for discovery purposes. Data holds can also be implemented for specific retention periods. Microsoft is positioning this type of hold to replace journaling and archiving solutions.

Management

For an on-premises Exchange deployment you need to perform the following management tasks:

  • Monitor performance
  • Monitor hardware for failures
  • Update firmware
  • Apply operating system updates
  • Apply Exchange Server updates
  • Manage mailboxes
On a system that is properly designed, there is minimal need to monitor performance (you built it with some head room), but there is an ongoing need to address issues such as failed indexes and large delivery queues. These issues are not relevant for Office 365 management.



As a cloud-based service, you are not responsible for any monitoring or updates in Office 365. Management of user mailboxes is still required.

New management tasks for Office 365 include:

  • Managing directory synchronization
  • Managing user licenses

Directory synchronization typically requires little management. There may be occasional troubleshooting, but it is rare. Generally once directory synchronization is configured, it just works.

Processes for creating new mailboxes do need to be modified to include assignment of Office 365 licenses. This can be scripted and run as a scheduled task if it is not required to be time sensitive.

If all users are migrated to Office 365 a local Exchange server is still maintained for management purposes. This is required because changes are made in the local AD and synchronized to Office 365. However, outages for this management server do not affect Office 365 users only management functionality. Microsoft provides a free license for this purpose.

Additional Features

Many Office 365 deployments begin as a replacement for on-premises Exchange. However, for mid-sized and large organizations, when you do a raw comparison of Exchange on-premises costs and Office 365 licensing costs, you may find that Exchange on-premises is actually a lower cost.

You need to consider that Office 365 includes more than just Exchange functionality. Depending on the licenses you select, there are various features. Also note, that you can mix and match licenses as required to give different features to different users to meet the needs of your organization.

For an accurate list of Office 365 plan features, see the Microsoft web site. However, here's a quick summary you can use to get a general idea of what's included with each license type:
  • Kiosk:
    • Exchange Online
    • SharePoint Online
  • E1: Kiosk features plus
    • One Drive for Business (1TB)
    • Skype for Business
    • Yammer
  • E3: E1 features plus
    • Office 365 Proplus (desktop office suite)
    • Azure Rights Management
  • E5: E3 features plus
    • Cloud PBX
These extra features provide more value than just Exchange online. I've been surprised at the number of larger organizations that have subscribed to Office 365 with E3 licenses as their preferred method of deploying Microsoft Office. Skype for Business is also a bit of a pain to implement on-premises and having that included with the E1 licenses is beneficial.