Automatic Updating ESXi Certificates

This is mostly a note to myself.

Venafi TPP can copy certificate files via SSH by using SCP. 

Look like ESXi supports this:

• I've always WinSCP'd into the host then gone to /etc/vmware/ssl/ rename and transfer the certificate as 'rui.crt' and rename and transfer the private key as 'rui.key' then go to the ESXi console, go to troubleshoot, and restart management agents. Your ESXi host should now be using the new certificates.
• https://www.reddit.com/r/vmware/comments/yenv8d/replace_esxi_host_ssl_with_internal_ca_with/ 

Other:

• https://knowledge.broadcom.com/external/article?legacyId=56441 
• https://www.filecloud.com/blog/2022/06/installing-an-ssl-certificate-on-an-esxi-server/
• https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-B70177EE-9C62-4BB9-BD3C-4848C128E63A.html
• https://www.starwindsoftware.com/blog/how-to-replace-your-default-esxi-ssl-certificate-with-a-self-signed-certificate/
• https://docs.vmware.com/en/VMware-Cloud-Foundation/4.5/vcf-admin/GUID-91824B56-91F3-44FE-B67B-180A50B94717.html#GUID-91824B56-91F3-44FE-B67B-180A50B94717
  

 

Determine Local SID via PowerShell

I recently needed to verify that SYSPREP had been run on a few servers. You can download a utility PSGETSID to do this, but I didn't want to introduce an external utility to the servers.

I found the following PowerShell code to get the SID of the local Administrator account.

$u = New-Object System.Security.Principal.NTAccount('administrator')
$SID = $u.Translate([System.Security.Principal.SecurityIdentifier])
$sid.Value

This returns a SID with -500 on the end. If you ignore the -500, the SID is the SID of the local machine. If this is unique, you know that SYSPREP was used.

You can also look for the CloneTag value in HKLM:\SYSTEM\Setup. This contains the time when SYSPREP was run. However, I prefer to see the unique SID.

More info about using SYSPREP:

• https://techcommunity.microsoft.com/t5/windows-blog-archive/the-machine-sid-duplication-myth-and-why-sysprep-matters/ba-p/723859
  

Duplicate OneDrive Icon in File Explorer

 I just got a new laptop and after doing most of the configuration I noticed that there were two OneDrive folders in File Explorer. The top one is named for my work tenant and the lower just named OneDrive. Typically the icon named OneDrive is for the personal version of OneDrive but I didn't have that configured. Only a OneDrive for Business account was configured.

When I clicked on the OneDrive icon it opened the same content as my OneDrive for Business. So, it appeared to redirect.

In my user profile folder (C:\Users\ByronWright\), I saw that there was a folder named OneDrive. When I deleted that folder, the redirection stopped and I got an error indicating that the folder didn't exist. Whoops. No fix there.

After a bit of searching around I found that I could remove it by removing the reference to it in the registry. That area of File Explorer is controlled by HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\Desktop\NameSpace. In that registry key, I had two registry keys that looked like GUIDs. Inside I could see one with the name OneDrive and the other was the OneDrive for Business name.

I deleted the registry key that referred to OneDrive (the entire key named like the GUID) and it immediately removed the extra reference to OneDrive from File Explorer.

My new laptop was originally purchased with Windows 11 Home edition which required me to sign in with a Microsoft account during setup. This configured personal OneDrive at the time. I upgraded the laptop to Windows 11 Pro and did a system reset but I think the reference to personal OneDrive was left over in the default user profile which was then copied when I signed in as my Azure AD (Entra ID) user.

For some additional information about this see: https://superuser.com/questions/1144868/duplicated-onedrive-icon-in-explorer

Immutable ID, ms-ds-consistencyGUID, and object GUID conversions

It seems like I'm constantly need to convert immutable IDs and GUIDs as part of my M365 migrations. To simplify this I finally got around to writing some functions that simplify the work instead of looking them up all the time.

I created the following:

• Convert-ImmutableIDToGUID
• Convert-ImmutableIDtoHexString
• Convert-ImmutableIDtoByteArray
• Convert-ByteArrayToGUID
• Convert-GUIDToImmutableID
• Convert-HexStringToGUID
• Convert-HexStringToImmutableID
• Convert-ByteArrayToImmutableID

To make these available at a powershell prompt, you can load them as part of your powershell profile or dot source a script that contains.

Example of dot sourcing:

. c:\scripts\convertfunctions.ps1

Code for the functions:

# Example immutable ID to play with
# $ImmutableID = "GJo33fsMIUKvmIIyTOSjzg=="

function Convert-ImmutableIDToGUID {
    param ($ImmutableID)
    $guid=[Guid]([Convert]::FromBase64String($ImmutableID))
    return $guid
}


function Convert-ImmutableIDtoHexString {
    param ($ImmutableID)
    $hexstring=([Convert]::FromBase64String($ImmutableID) | ForEach-Object ToString X2) -join ' '
    return $hexstring
}


function Convert-ImmutableIDtoByteArray {
    param ($ImmutableID)
    $bytearray=[Convert]::FromBase64String($ImmutableID)
    return $bytearray
}


#When you retrieve ms-ds-consistencyGUID from AD it is a byte array
#to avoid this conversion use [guid]$user.'ms-ds-consistencyGUID'
function Convert-ByteArrayToGUID {
    param ($bytearray)
    $guid=[Guid]([Convert]::FromBase64String([system.convert]::ToBase64String($bytearray)))
    return $guid
}

#works with GUID object or string in GUID format
function Convert-GUIDToImmutableID {
    param ($guid)
    $immutableID = [system.convert]::ToBase64String(([GUID]$guid).ToByteArray())
    return $immutableID
}


function Convert-HexStringToGUID {
    param ($hexstring)
    $guid = [GUID]([byte[]] (-split (($hexstring -replace " ", "") -replace '..', '0x$& ')))
    return $guid
}


function Convert-HexStringToImmutableID {
    param ($hexstring)
    $ImmutableID = [system.convert]::ToBase64String([byte[]] (-split (($hexstring -replace " ", "") -replace '..', '0x$& ')))
    return $ImmutableID
}


function Convert-ByteArrayToImmutableID {
    param ($bytearray)
    $ImmutableID = [system.convert]::ToBase64String($bytearray)
    return $ImmutableID
}

<# Use Example

$msdsconsistencyGUID = (Get-ADUser Byron -properties *).ms-ds-consistencyGUID
$ImmutableID = Convert-ByteArrayToImmutableID -bytearray $msdsconsistencyGUID
Set-AzureADUser byron@domain.com -ImmutableID $ImmutableID

#>

Using a specific version of Exchange ECP

When you have multiple versions of Exchange Server installed during a migration, you might need to specify which version of ECP you want to use. For example, migrating for Exchange 2013 to Exchange 2019 you might want to use one or the other. By default, when you sign in, you are directed to the version where your mailbox resides.

For users without mailboxes, they are redirected to the version of Exchange where the arbitration mailboxes are located. So, it's generally best to move the arbitration mailboxes to the newest version of Exchange server early in your migration process.

When you access ECP, you can specify which version of ECP you want to use as part of the URL by adding ?ExchClientVerXX. For example, https://mail.contoso.com/ecp?ExchClientVer=15.

You can use the following version numbers:

• 14 - Exchange 2010
• 15 - Exchange 2013
• 15.1 - Exchange 2016
• 15.2 - Exchange 2019

 References:

• https://learn.microsoft.com/en-us/answers/questions/578660/exchange-2019-ecp-redirects-to-exchange-2013
• https://learn.microsoft.com/en-us/exchange/troubleshoot/client-connectivity/exchange-server-2013-2016-redirect-to-2010-owa-ecp
  

DMARC, DKIM, and SPF for email

There is a huge amount of spam and phishing emails on the internet. To combat this recipients are getting more and more aggressive in trying to block it. As a mail system administrator, you want the valid email from your domain to be accepted by recipients. 

There are three different (but related) technologies that you should be aware of that validate email from your domain:

• SPF (Sender Policy Framework)
  
• DKIM (DomainKeys Identified Mail)
  
• DMARC (Domain Message Authentication Reporting and Conformance)
  

All of these technologies are used by recipients to validate that email is sent from an authorized email system. Correctly configuring these technologies increase the likelihood that email you send won't be incorrectly categorized as spam.

These technologies are also useful to prevent spoofing email from entering your email system. Many phishing attempts try to appear as though mail comes from your domain. SPF and DKIM help identify these. For example, a message from it@contoso.com would be identified as unauthorized when coming from a random IP address on the internet rather than your own authorized servers. 

SPF

An SPF record is a TXT record in DNS that list authorized sources for email in your domain. For example, the SPF record in contoso.com defines which servers are authorized to send email from the contoso.com domain. You need to identify all of the services that send email on behalf of your domain before you configure the SPF record. Valid sources can be applications or marketing services in addition to your email provider.

The value in the TXT record contains host names (a) and IP addresses (ip4 or ip6) that are allowed to send email for your domain. If you have three servers in your organization that send email to the internet, then you need to include the IP addresses or host names for those servers in your SPF record. Remember that the IP address and host name are from the perspective of hosts on the internet. You need to include host names that can be resolved by internet DNS or IP addresses that are routable on the internet.

If you're using a hosting service, such as Microsoft 365 or Gmail, to send messages, they'll provide instructions on how to configure SPF for their service. In many cases, they maintain lists of host names and IP addresses so you don't have to. In your SPF record, you refer to their configuration by using the include option.

Example:

• Type: TXT
• Name: @
  
• Value: v=spf1 include:spf.protection.outlook.com ip4:1.1.1.1 -all

Description of record contents:

• All SPF records start v=spf1 to indicate that they are an SPF record.
• The include option in this example is used for Microsoft 365. Microsoft maintains the detailed information required at this DNS location.
• The ip4 option specifies that the server 1.1.1.1 is allowed to send email on behalf of the domain.
• The all option is always includes at the end of the SPF record. When it's configured with a dash (-), then mail not from a valid host should be a hard fail. When it's configured with a tilde (~), then mail not from a valid host should be a soft fail.

 Additional resources:

• How to Create and Add an SPF Record
  https://dmarcian.com/create-spf-record/
• SPF Record Syntax
  https://dmarcian.com/spf-syntax-table/
• SPF Record Check
  https://mxtoolbox.com/spf.aspx

DKIM

DKIM adds a digital signature to email messages to verify that they are from an authorized source. The recpient email system reads the digital signature and verifies that it's valid for your domain. The digital signature is validated based on DNS records that you publish.

The process for implementing DKIM varies depending on the email provider. Microsoft has an easy to use system in Exchange Online where they manage most of the DNS work for you. Other vendors provide information for you to create your own DKIM DNS records. And, some vendors and applications don't support DKIM at all. For example, on-premises Microsoft Exchange Server doesn't have DKIM functionality included.

In Microsoft 365 (Exchange Online) the DKIM process is pretty simple to implement. You need to create two CNAME records in your domain that point to records created by Microsoft and then turn on DKIM for the domain. The following example record is for contoso.com:

• Type: CNAME
• Name: selector1._domainkey
• Value: selector1-contoso-com._domainkey.contoso.onmicrosoft.com

When an email that's signed using DKIM is received, it includes a digital signature. The digital signature refers to the selector that contains the key used to used validate the signature. The DNS example above is for selector1 in the contoso.com domain. The CNAME record directs recipients to selector1-contoso-com._domainkey.contoso.onmicrosoft.com to obtain the necessary key for validation.

Additional resources:

• What is DKIM?
  https://dmarcian.com/what-is-dkim/
• Help prevent spoofing and spam with DKIM (Google)
  https://support.google.com/a/answer/174124?hl=en
• Use DKIM to validate outbound email sent from your custom domain (Exchange Online)
  https://learn.microsoft.com/en-us/microsoft-365/security/office-365-security/email-authentication-dkim-configure?view=o365-worldwide

DMARC

A DMARC record is a TXT record in DNS that tells recipients two main things about email from your domain:

• How to handle email that fails validity checks.
• Where to send reports about email that fails validity checks.

Email is validated when it passes either the SPF check or the DKIM check. This means that you can begin using DMARC when you have implemented SPF but don't have DKIM. This is important since some email systems don't support DKIM.

The p option in the record allows to you define what action should be taken on messages that fail SPF and DKIM. The potential values are:

• none - Specifies that no action should be taken and messages that don't pass should be delivered. You can use this during initial setup when you're still validating your configuration.
• quarantine - Specifies that unvalidated messages should be send to quarantine or junk email. You can use this when you think your setup is configured correctly but still want more time to validate.
• reject - Specifies that unvalidated messages should be discarded. You should use this option when you configuration is fully validated and you know that your SPF and DKIM are configured properly.

An important benefit of DMARC is the ability to define an email address for reporting messages that were not validated. A recipient email system that receives in invalid email sends a report to the email address. Then you can review the reports and identify any valid systems or applications sending email from your domain and fix the configuration. This email address is defined by the rua option.

For a busy email system, it's not realistic to review all of the DMARC reports manually. Instead, there are many services such as dmarcian (dmarcian.com) that will host a mailbox for you and process the DMARC reports into usable information.

Example DMARC record:

• Type: TXT
• Name: _dmarc
• Value: v=DMARC1; p=reject; rua=mailto:dmarcreports@contoso.com;

 Description of record contents:

• The DMARC record for a domain always has the name _dmarc.
• A DMARC record always starts with v=DMARC1.
• The p option defines the action for unvalidated messages.
• The rua option defines the email address for reports.
• The  semicolon (;) at the end is optional.

Additional resources:

• DMARC Overview
  https://dmarc.org/overview/
• DMARC Record Wizard
  https://dmarcian.com/dmarc-record-wizard/

Summary

Every organization should have SPF and DMARC in place. You can do this without having DKIM enabled because the SPF passing is sufficient. However, DKIM is an additional assurance of validity for email delivery and you should implement it where you can.

Be aware that receiving email systems can do whatever they want with inbound mail for spam filtering. So, it's possible that even if you have SPF, DKIM, and DMARC configured correctly, a message can be blocked, but it's much less likely.

 

Cannot ping the selected CA when renewing certificate

I've configured a test environment for migrating certification authorities (CAs) from Windows Server 2008 R2 to Windows Server 2019. The process is well documented and I didn't expect any issues. However, after migrating the root CA to Windows Server 2019, I couldn't renew the CA certificate on the issuing CA still running on Windows Server 2008 R2.

Attempting to renew the certificate I got this error:

Cannot ping the selected CA. Please make sure the CA is running. The RPC server is unavailable. 0x800706ba (WIN32: 1722)

On receiving this error, I tried the common things like verifying network connectivity and disabling Windows Firewall just in case. Some searching around indicated this is sometimes a result of COM+ permissions, but it looked OK. Eventually I found this event on the Windows Server 2019 CA which led me to the source of the error.

Event information:

• Log Name: System
• Source: DistributedCOM
• Event ID: 10036
• Description: The server-side authentication level policy does not allow the user CERTTEST\Administrator SID (S-1-5-21-2892548479-535578393-614425194-500) from address 192.168.0.112 to activate DCOM server. Please raise the activation authentication level at least to RPC_C_AUTHN_LEVEL_PKT_INTEGRITY in client application.
  

The actual cause is DCOM security hardening introduced by Microsoft in KB5004442. This update requires a higher level of security that Windows Server 2008 R2 doesn't support. Until March 2023 you can disable the security hardening with a registry key, but after that you can't downgrade the security.

• Path : HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Ole\AppCompat
• Value Name: "RequireIntegrityActivationAuthenticationLevel"
• Type: dword
• Value Data: default = 0x00000000 means disabled. 0x00000001 means enabled. If this value is not defined, it will default to enabled.
  

For applications that are not updated to request the higher level of authentication security, Microsoft has added new DCOM client functionality to automatically raise the security level in the November 2022 update. This should mitigate issues for most client applications on Windows Server 2012 R2 and newer or Windows 8.1 and newer.

For detailed information about KB5004442 see:

• https://support.microsoft.com/en-us/topic/kb5004442-manage-changes-for-windows-dcom-server-security-feature-bypass-cve-2021-26414-f1400b52-c141-43d2-941e-37ed901c769c