OAuth Certificates with Hybrid Exchange

Older versions of Microsoft Exchange in a hybrid configuration with Exchange Online (EXO) used a federation trust to authenticate connections for free/busy information. Newer hybrid deployments of Exchange 2016/2019 use OAuth authentication instead of federation.

OAuth authentication is reliant on the Auth certificate in your on-premises Exchange. This certificate is created automatically with a lifetime of 5 years when you install Exchange Server on-premises. If this certificate has been replaced, then you also need to update Azure AD with the new certificate information. The simplest way to update the information is by running the hybrid wizard again after you update the Auth certificate.

I wrote a previous post about renewing/updating the Exchange Server Auth certificate here:

• http://byronwright.blogspot.com/2018/05/expired-microsoft-exchange-server-auth.html
  

If you update the Exchange Server Auth certificate and forget to update the information in Azure AD, you are likely to see free/busy lookups to EXO fail. I recently saw this as a client and decided to dig into the configuration a little bit more.

Testing OAuth Connectivity

You can test OAuth authentication from the Exchange Management Shell on-premises. When doing this, you need specify a local mailbox with the following command:

Test-OAuthConnectivity -Service EWS -TargetUri 'https://outlook.office365.com/ews/exchange.asmx' -Mailbox user@domain.com -Verbose | FL

With the FL in the above command you'll see detailed information returned. If you run the command without FL and it's successful, you'll see output like this:

 

If the test is unsuccessful, you'll see text something like:

• The remote server returned an error: (401) Unauthorized
  
• Unable to get the token from Auth Server
• Client assertation contains an invalid signature [Reason - The key was not found]
  

Verifying the Certificate Used for OAuth

To identify the Auth certificate currently used by on-premises Exchange Server, run Get-AuthConfig. In the example below, you can see the thumbprint for the currently used certificate. If the Auth certificate had been updated, it may show a previous certificate thumbprint too. The service name returned by this command is a GUID for Exchange Online.

Once you have the thumbprint of the Auth certificate, you can verify it exists on each of the on-premises Exchange servers. All servers should have the same Auth certificate. Use the following command for each of your on-premises Exchange servers.

Get-ExchangeCertificate ThumbprintFromAuthConfig -Server ServerName

Verifying Certificate Information in Azure AD

The Auth certificate configured in on-premises Exchange Server is used for client authentication to Azure AD for free/busy lookups. The public portion of the certificate is stored in Azure AD for this purpose. You can view this information by using the MSOL or AzureAD cmdlets.

To view the certificate information with the MSOL cmdlets, run the following command:

Get-MsolServicePrincipalCredential -ServicePrincipalName "00000002-0000-0ff1-ce00-000000000000" -ReturnKeyValues $true

 

This command may return multiple results. In my test environment, there are multiple entries for the same certificate. I assume that this is because it gets added each time I run the hybrid wizard.

The easiest way to verify that the certificate in Azure AD matches your on-premises Auth certificate is by using the StartDate and EndDate fields. These will match the NotBefore and NotAfter properties shown by Get-ExchangeCertificate. Be aware that StartDate and EndDate are in UTC time and the NotBefore and NotAfter are probably showing in your local time zone.

You can also save the Value property in a text file with the extension .cer. Then you can open the .cer file and view the certificate information. This includes additional information such as the Thumbprint which you can verify against the on-premises Auth certificate.

 

You can use the Azure AD cmdlets to get similar information (but not the certificate value). This is a bit more complex and requires multiple steps.

To get the ObjectID of the SPN for Exchange online:

$spnID = (Get-AzureADServicePrincipal | Where DisplayName -eq 'Office 365 Exchange Online').ObjectID

 

To list the certificates that have been uploaded:

$certs = Get-AzureADServicePrincipalKeyCredential -ObjectID $spnID

 

To get the thumbprint of the most recent certificate uploaded:

[system.convert]::ToBase64String($certs[0].CustomKeyIdentifier)

 

 Update Auth Certificate in Azure AD

If you find that the on-premises auth certificate is not present in Azure AD, the best solution is running the hybrid wizard again. Running the hybrid wizard will update the OAuth certificate.

If for some reason you don't want to run the hybrid wizard, you can update the certificate manually by exporting it from on-premises and importing it into Azure AD.

Steps 3 and 4 in the following document describe how to manually export and import the Auth certificate:

• https://docs.microsoft.com/en-us/exchange/configure-oauth-authentication-between-exchange-and-exchange-online-organizations-exchange-2013-help#step-3-export-the-on-premises-authorization-certificate

Links found during research

• https://newsignature.com/articles/on-premises-mailbox-cannot-see-free-busy-information-for-exchange-online-mailbox/
• https://social.technet.microsoft.com/Forums/en-US/a0bb1d0a-6639-4fdf-92e1-7fa350e47057/exchange-2016-onpremises-users-are-not-able-to-view-calendar-information-of-the-office-365-users?forum=exchangesvrdevelopment
• https://techcommunity.microsoft.com/t5/exchange-team-blog/demystifying-hybrid-free-busy-finding-errors-and-troubleshooting/ba-p/607727
• https://techcommunity.microsoft.com/legacyfs/online/media/2019/01/FB_Errors.FixesV6.pdf

 

Laggy Mouse and Jaggy Fonts

I have a 4K TV hooked up as my monitor with an older video card. Unfortunately, this video card can only output 4K at 30Hz which isn't optimal, but for my purposes is just fine. I'm not playing games that require fine tuned actions.

I recently changed my display to 1080P for an online presentation and then back to 4K. By default, 4K was running at 60Hz, however, this made the text slightly blurry because HDMI was compressing the signal. Set it down to 30Hz to fix that, but then the text was jaggy and not smooth. Also, my mouse was really laggy as I moved it around the screen.

When I changed back to 4K 30Hz, Windows 10 and the TV negotiated using HDR (High Dynamic Range) for display. Normally when Windows 10 negotiates a setting, that's preferred, but not in this case. When I disabled HDR in display settings, my fonts were smoothed properly and the mouse lagging stopped.

Convert ImmutableID to Hex for AD

To get the immutableID value from a user (should be able to do similar with Get-MSOLUser if preferred):

$id = (Get-AzureADUser -ObjectId User@domain.com).immutableid

To convert that ID to hex for entry: 

$hex=([system.convert]::FromBase64String("$id") | ForEach-Object ToString X2) -join ' '

To view the value in $hex:

$hex

The immutable id will be a value something like: fhG+Kox7LkaYwSIf6s6UFA==

The hex for that one is: 7E 11 BE 2A 8C 7B 2E 46 98 C1 22 1F EA CE 94 14

The hex value can be entered into the ms-DS-ConsistencyGUID attribute of the user object.

And converting from objectGUID to ImmutableID:

$immutableID = [system.convert]::ToBase64String(([GUID]($u.ObjectGUID)).tobytearray())

And converting ImmutableID to GUID:  

$objectGUID = [Guid]([Convert]::FromBase64String($ImmutableID))

 

UPDATE: I've created a set of functions that you can use for conversions at https://byronwright.blogspot.com/2023/08/immutable-id-ms-ds-consistencyguid-and.html

Install-Module Fails without TLS 1.2

I've run into problems with Windows Server where the Install-Module cmdlet generate errors and won't download from the PowerShell  repository on the internet. To fix this you need to enable TLS 1.2 for PowerShell.

WARNING: Unable to resolve package source 'https://www.powershellgallery.com/api/v2/'.
PackageManagement\Install-Package : No match was found for the specified search
criteria and module name 'xxxxxxxxx'. Try Get-PSRepository to see all available 
registered module repositories.

To do this permanently for .NET 4 and up, set two registry keys for 64-bit and 32-bit .NET Framework:

Set-ItemProperty -Path 'HKLM:\SOFTWARE\Wow6432Node\Microsoft\.NetFramework\v4.0.30319'-Name 'SchUseStrongCrypto' -Value '1' -Type DWord
Set-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\.NetFramework\v4.0.30319' -Name 'SchUseStrongCrypto' -Value '1' -Type DWord

If you need to do a quick temporary fix because you can't update the registry then use this:

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

The temporary fix is only for the current PowerShell prompt.

Issues with Exchange 2010 and Exchange 2016 coexistence

This one is primarily notes to myself...

• When Exchange 2016 is installed with Exchange 2010, MAPI over HTTP is enabled by default for the organization.
• Exchange 2010 mailboxes continue to accessed via RPC.
• Exchange 2016 mailboxes will use MAPI over HTTP
• If Exchange 2016 mailboxes have Full Access to an Exchange 2010 mailbox then Outlook Anywhere is used to connect to that secondary mailbox.
• If using a wildcard cert you need to set the certificate name for the EXPR outlook provider for Outlook Anywhere as msstd:*domain.com.

Authentication prompts

Exchange 2010 on Windows Server 2008 R2 requires a security update for Outlook Anywhere to function properly. This is a security update from 2016 (KB3140410). It "should" already be in place, but if it's not then Outlook Anywhere will cause tons of authentication popups in Outlook.

• https://support.microsoft.com/en-us/help/2990117/outlook-anywhere-users-prompted-for-credentials-when-they-try-to-conne

I saw this manifest as Exchange 2016 mailboxes with a secondary mailbox on Exchange 2010 getting the popups. Only an Exchange 2016 mailbox was fine because it used only MAPI over HTTP on  Exchange 2016. Only an Exchange 2010 mailbox was fine because it used only RPC to Exchange 2010.

If the update is not in place and you don't have the opportunity to apply the update quickly, you can modify DefaultAppPool in IIS Manager to use the identity Network Service. Recycle DefaultAppPool for the the change to take effect. Recycling DefaultAppPool does not affect users.

Error message that you will likely see in RpcHTTP proxy log (C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\RpcHttp):

Complete=PrepareServerRequest;,WebExceptionStatus=ProtocolError;ResponseStatusCode= 401; WebException=System.Net.WebException: The remote server returned an error: (401) Unauthorized. at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) at
Microsoft.Exchange.HttpProxy.RpcHttpProxyRequestHandler.c__DisplayClass1.nullb__0(); HttpException=System.Web.HttpException (0x80004005): NegotiateSecurityContext failed with for host 'mail.contoso.com' with status 'InvalidToken' at Microsoft.Exchange.HttpProxy.KerberosUtilities.GenerateKerberosAuthHeader.

Win7 certificate errors

Windows 7 clients that don't have TLS 1.1 and 1.2 enabled might see a certificate error when connecting to Exchange 2016 for web services (not necessarily mailbox). To enable TLS 1.1 and 1.2 on Windows 7, you need to ensure that update KB3140245 is installed. With the update installed, you need to create additional registry entries.

• https://support.microsoft.com/en-ca/help/3140245/update-to-enable-tls-1-1-and-tls-1-2-as-default-secure-protocols-in-wi

The registry keys created by the quickfix utility distributed with this update by Microsoft are:

• HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\WinHttp
• Create DWORD: DefaultSecureProtocols
• Value: 0xA00
• HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Internet Settings\WinHttp
• Create DWORD: DefaultSecureProtocols
• Value: 0xA00

• HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings
• Create DWORD: SecureProtocols
• Value: 0xA8
• HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings
• Create DWORD: SecureProtocols
• Value: 0xA8

The DefaultSecureProtocols key is used by the Office Apps and the value 0xA00 designates TLS 1.1 and TLS 1.2.

The SecureProtocols key is used by Internet Explorer and the value 0xA08 designates TLS 1.0, TLS 1.1, and TLS 1.2.

MIS2000 Links

Using Power BI for data analytics and reporting
https://docs.microsoft.com/en-us/learn/paths/create-use-analytics-reports-power-bi/

City of Winnipeg Software Piracy
https://www.cbc.ca/news/canada/manitoba/city-of-winnipeg-manager-in-charge-of-police-radios-arrested-after-2-year-investigation-1.5027975

What's wrong with this picture?
https://twitter.com/DaveLeeBBC/status/1102359402151985152

Computer system failure grounds transit system in San Francisco
https://www.bart.gov/news/articles/2019/news20190309

Supply Chain Management Simulator
https://www.scmglobe.com/supply-chains-roman-empire/

Career wisdom from IT pros
https://www.reddit.com/r/sysadmin/comments/dzm3xs/once_a_young_sysadmin_and_now_an_old_unicorn_how/

(A few) Ops Lessons We All Learn The Hard Way
https://www.netmeister.org/blog/ops-lessons.html

Michael Geist blog (copyright and net freedom issues)
http://www.michaelgeist.ca/

Government IT failures

Federal Government - Phoenix payroll system #1
http://www.oag-bvg.gc.ca/internet/English/parl_oag_201711_01_e_42666.html

Federal Government - Phoenix payroll system #2
http://www.oag-bvg.gc.ca/internet/English/parl_oag_201805_01_e_43033.html

More Phoenix payroll
https://twitter.com/PSSuzanne/status/1098720582516895745

Phoenix payroll - 3 years later and still broken
https://www.thestar.com/news/canada/2019/03/10/still-stuck-on-the-phoenix-pay-roller-coaster-these-canadians-just-want-to-get-off.html

Federal Government - Gun registry
https://www.cbc.ca/news/canada/timeline-the-gun-registry-debate-1.786548

Security

Malware distribution hosted in LED light control console
https://news.microsoft.com/apac/features/law-enforcement-and-microsoft-come-together-to-bust-a-major-malware-attack-in-taiwan/

Ransomware attacks lock 2 Manitoba law firms out of computer systems

https://www.cbc.ca/news/canada/manitoba/winnipeg-law-firms-computer-virus-ransomware-1.5530825

Is Huawei really a risk?
https://www.reuters.com/article/us-huawei-tech-usa-pompeo/u-s-wont-partner-with-countries-that-use-huawei-systems-pompeo-idUSKCN1QA1O6?utm_source=reddit.com

City of Saskatoon phishing
https://thestarphoenix.com/opinion/columnists/tank-scam-stings-at-saskatoon-city-hall-with-shiny-fiscal-reputation

City of Ottawa victim of phishing
https://www.ctvnews.ca/canada/ottawa-city-treasurer-transfers-130k-of-taxpayer-funds-to-email-fraudsters-1.4371900

Stolen laptop with health data
https://www.cbc.ca/news/canada/north/nwt-stolen-laptop-encryption-it-department-1.5044118

Government of Nunavut Ransomware
https://nunatsiaq.com/stories/article/government-of-nunavut-slowly-rebuilds-computer-network-following-ransomware-attack/

Nursing Home Network Ransomware
https://www.cbsnews.com/news/hackers-ransomware-nursing-homes-14-million/

Spear Phishing
https://www.microsoft.com/security/blog/2019/12/02/spear-phishing-campaigns-sharper-than-you-think/

Scammy companies

WeWork - A tech company?
https://www.businessinsider.com/weworks-nightmare-ipo?r=US&IR=T?utm_source=markets&utm_medium=ingest

WeWTF
https://www.profgalloway.com/wewtf

Theranos timeline
https://www.refinery29.com/en-ca/2019/01/223033/theranos-scandal-timeline-what-happened-elizabeth-holmes-documentary

Theranos - Wall Street Journal expose
https://www.wsj.com/articles/theranos-has-struggled-with-blood-tests-1444881901

Cryptocurrency

What is cryptocurrency?
https://blockgeeks.com/guides/what-is-cryptocurrency/

QuadrigaCX Cryptocurrency Exchange Shadiness
https://www.cbc.ca/news/business/quadriga-bitcoin-bankruptcy-1.5004735

QuadrigaCX empty accounts - even more shady
https://www.ctvnews.ca/business/search-into-missing-cryptocurrency-turns-up-empty-cold-wallets-report-1.4319270

Cryptoqueen: How this woman scammed the world, then vanished
https://www.bbc.com/news/stories-50435014

Azure AD Connect Large Object Error

A client is migrating their remaining mailboxes from on-premises Exchange to Office 365. Today they went to migrate a mailbox, but the user account wasn't replicated up to Office 365. After verifying that it was not being filtered by OU in Azure AD Connect, I checked the Synchronization Service Manager for Azure AD Connect and found an error listed for the export to the Azure AD tenant (XXX.onmicrosoft.com).

The error was LargeObject and when I drilled down, it had these details:

The provisioned object is too large. Trim the number of attribute values on this object.

This error is typically caused by:

• Too many user certificates (15 max)
• Too many SMIME certificates (15 max)
• A thumbnail photo that is too large
• Too many proxy addresses

This user object did not have any user certificates, SMIME certificates, or a thumbnail photo. So, let's check out the proxy addresses.

The user object had 540 addresses. After a bit more research, I found that user objects in Azure AD have a limit of 400 proxy addresses, Azure AD Connect has a limit of 333 proxy addresses.

They do have a legitimate need for this account to receive mail for all of those addresses. We implemented a workaround by creating a group for the extra addresses. We removed 300 email addresses and put them on a group where that user is the only member. Mail flow is preserved and now both the user and the group can sync. The group is hidden from address lists to avoid confusing the users.

More information:

• https://docs.microsoft.com/en-us/azure/active-directory/hybrid/tshoot-connect-sync-errors#largeobject
• https://feedback.azure.com/forums/169401-azure-active-directory/suggestions/38101924-aad-connect-enhance-large-object