Tuesday, June 1, 2021

DNS Forwarding Timeouts

When you configure forwarders on Windows DNS servers, it's not obvious what the timeout values are. You might intuitively think that more forwarders is better. In reality, with the default values, you're just kidding yourself.

DNS forwarders have a default timeout of 3 seconds. If the first forwarder does not respond within 3 seconds then the second forwarder is contacted, and so forth.

However, there is an overall recursion timeout of 8 seconds. After 8 seconds no more forwarders will be contacted. So, best case, the process looks like this:

  • 0s - Contact forwarder 1
  • 3s - Contact forwarder 2
  • 6s - Contact forwarder 3
  • 8s - recursion timeout (process ends)

As you can see, only the first three forwarders listed are ever used. Putting more than 3 forwarders on a DNS server is misleading because forwarders 4 and up will never be contacted.

Conditional forwarders have a similar process but with different timeout values. Conditional forwarders have a default timeout of 5 seconds along with the recursion timeout of 8 seconds. This means that only two conditional forwarders are ever contacted.

The conditional forwarder process looks like this:

  • 0s - Contact conditional forwarder 1
  • 5s - Contact conditional forwarder 2
  • 8s - recursion timeout (process ends)

Again, I suggest don't ever list more than two conditional forwarders or it is misleading.

If you want to allow additional forwarders or conditional forwarders to be queried, you can modify the default values in the registry of the DNS servers. However, be sure to do this on all DNS servers so that it is consistent. And, document it as part of your domain controller build process so that it is configured on new domain controllers too.

Registry keys to modify the default timeout values:

  • Recursion timeout (per DNS server)
    • HKLM\SYSTEM\CurrentControlSet\Services\DNS\Parameters\**RecursionTimeout
  • Forwarding timeout (per DNS server)
    • HKLM\SYSTEM\CurrentControlSet\Services\DNS\Parameters\**ForwardingTimeout
  • Forwarder timeout (per zone/conditional forwarder)
    •  HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\DNS Server\Zones\ <zone_name>\ForwarderTimeout

For more detailed information about this process, see:

Wednesday, March 24, 2021

Script to compare file presence in directory structures

I'm still running into a few organizations that need to convert their sysvol replication from FRS to DFS-R. The process for doing that is quite straight forward, but if FRS replication is broken, then there is the risk that some data could be lost during this process.

As part of due diligence before converting, I like to compare the contents of sysvol among the domain controllers. The PDC emulator is the default source for new sysvol data. So, I compare that to other domain controllers. To help me with this I've created a PowerShell script that compares the files to identify any that are not the same on two servers.

I use this for sysvol, but you could use it to compare any two data structures. This script does not compare time stamps, file size, or file contents. It only looks for presence.


#Domain controller path to sysvol

#Get list of files
$srcfiles = get-childitem -Path $srcpath -File -Recurse
$targetfiles = get-childitem -Path $targetpath -File -Recurse

#Add property looks only at relative file name path
#Required for proper comparison without server name
Foreach ($file in $srcfiles) {

    $cleanpath = ($file.FullName).replace($srcpath,"")
    $file | Add-Member -NotePropertyName ShortPath -NotePropertyValue $cleanpath -Force


Foreach ($file in $targetfiles) {

    $cleanpath = ($file.FullName).replace($targetpath,"")
    $file | Add-Member -NotePropertyName ShortPath -NotePropertyValue $cleanpath -Force


$dif = Compare-Object $srcfiles $targetfiles -property ShortPath #-PassThru

Write-Host "Source ( <= ) is: " $srcpath
Write-Host "Target ( => ) is: " $targetpath
Write-Host ""

Tuesday, March 23, 2021

0x80070780: The file cannot be accessed by the system

Today got this error when trying to import a virtual machine on my newly rebuilt Hyper-V VM host running Windows Server 2019:

0x80070780: The file cannot be accessed by the system

I tried playing with permissions but it kept erroring out saying that it couldn't access the vhdx files. Finally, I realized that when I rebuilt the VM host, I forgot to install the data deduplication feature and thus the system could see the file names, but not access the deduplicated data.

After I enabled the data deduplication role service and rebooted, the system recognized the drives had deduplicated data and could access the files without issue.

Thursday, February 25, 2021

Synchronize membership from an AD group to a Microsoft 365 group

A client recently wanted to control access to Microsoft Stream content by using members of an AD security group that is already synchronized into Azure AD. However, Microsoft Stream access can only be controlled by using Microsoft 365 groups (formerly Office 365 groups). This led to identifying how we can synchronize the group membership from the AD group to the Microsoft 365 group.

The first option I thought of was Windows PowerShell. This is certainly possible, but it would need to be run as a scheduled task with credentials stored securely. Have reliance on an on-premises scheduled task was suboptimal. So, I went searching and found a template name Synchronize an Azure AD Group with an Office 365 Group on a recurring basis in Power Automate (formerly Flow) that is for exactly this purpose.


At a high level, this is what the flow does:

  • Sets a schedule for running
  • Queries membership from a source group
  • Queries membership from a target group
  • Compares the source and target group membership
  • Adds source members members not present in the target group
  • Sends a notification email identifying source members that were added
  • Identifies target members that are not present in the source group (with the option to provide and exceptions list of target members not to remove)
  • Sends an approval request to remove target members that are not present in the source group
  • Removes target members that are not present in the source group when the request is approved

When you add the template it will prompt you for permissions to create connections required to run the flow. The Office 365 Groups and Azure AD connectors require credentials to query and modify group memberships.

After you've created the flow from the template there are a few items that you need to configure in the flow. The Recurrence box defines how often the flow runs. The example below is configured to run the flow daily at 2am.

Next you need to configure the SourceGroupID and the TargetGroupID. You need to obtain the Object Id attribute for these groups from either the Azure AD admin center or Windows PowerShell. The Object ID for the group is placed in the Value box. There are separate steps for SourceGroupID and TargetGroupID. Effectively, these are populating variables used later in the script.

You also need to define the ApprovedOwnerUPN. This user approves removals from the target group when necessary. Enter the UPN for that user into the Value box.

Members in the target group that are not members in the source group are removed. If there are some unique members that you want to remain in the target group, you can define them in the ExcludedFromRemove variable. This might be useful for cloud only users such as an admin account.

To define the user accounts that are not removed, click on the createArray function in the Value box. This opens a box where you can manually enter the users you don't want to remove. In the screenshot below, susan@contoso.com and jeff@contoso.com are excluded from removal.

The List source group members and List target group members steps allow you to define a maximum number of results that are returned. By default, this value is 500. You need to define this value large enough to gather the entire membership of each group. Enter the number of group members in the Top box. If you leave this value blank, only 100 results are returned.

If your groups have membership higher than 1000 then you need to perform additional configuration to allow more than 1000 results. Click the ellipsis in the List source group members step and then select Settings to get the following screen. To allow more than 1000 results you need to turn on pagination and specify the number of results you want to allow. The example below allows up to 2000 members. After you've done this, the Top value is ignored.

The only other issue I had when using this flow was related to the officeLocation attribute of the user objects in Azure AD. This attribute correlates with the physicalDeliveryOfficeName attribute in on-premises AD and is visible as Office on the General tab of a user in Active Directory Users and Computers.

If the officeLocation attribute is blank, then the flow fails. In the environments I deal with, this value is often blank. Rather than putting in dummy office information, we can edit the flow to ignore the null value. 

The two affected steps are:

  • Parse UsersAdded values in Send mail if UsersAdded variable is not empty
  • Parse MembersToRemove values in Get approval if UsersToRemove is not empty

Both of these steps have a Schema box that defines how attributes are selected from earlier data. The officeLocation attribute is defined as a string and thus fails when there is a null value. A quick way to fix this problem is by removing officeLocation from the schema. Remove the three lines are highlighted from both steps.

An alternative that I haven't tested is modifying the acceptable values for officeLocation. You can see that above officeLocation, the mobilePhone attribute allows the type to be either string or null. I believe that syntax would also work for officeLocation.

For more information about Power Automate (formerly Flow), see:

Wednesday, February 24, 2021

Cisco AnyConnect blocked in Hyper-V virtual machine

Cisco AnyConnect is popular VPN software. The VPN server can enforce policies on the connecting clients. One control is blocking access from remote desktop connections. I assume that this is primarily to block connections from Remote Desktop servers or Windows 10 Remote Desktop where the same computer might be simultaneously shared by multiple users.

The error message you'll see is:

VPN establishment capability from a remote desktop is disabled. A VPN connection will not be established.

You might see this error when you use Hyper-V Manager to access a virtual machine and run Cisco AnyConnect. By default, the connection to a virtual machine is an enhanced session that is based on RDP. If you disable the Enhanced session setting in the View menu then Cisco AnyConnect will run and connect properly.

What is Azure AD Domain Services?

Sometimes letting go of old information and assumptions is the hardest part of learning something new. That's what I ran into today trying to wrap my head around Azure AD Domain Services. To purge your brain, let me start by saying that Azure AD Domain Services does not behave like an off site domain controller for your on-premises deployment of Active Directory Domain Services (AD DS).

Some quick definitions:

  • Active Directory Domain Services (AD DS) - Commonly called Active Directory, this is your local directory service/domain. You have domain controllers for this domain. This domain holds user and computer objects.
  • Azure Active Directory - Commonly called Azure AD, this is the cloud directory service used for Microsoft cloud services such as Exchange Online and SharePoint Online.
  • Azure AD Connect - This is software that you run on-premises to synchronize users and groups from AD DS on-premises to Azure AD. This allows your users to sign in to Microsoft cloud services by using the same username and password as the local AD.

Azure AD Domain Services (Azure AD DS) is a limited version AD DS that is provided as a cloud service. Like AD DS, it can have user and computer accounts.

The main use case for Azure AD DS is hosting line of business applications in Azure. The virtual machines in Azure can be joined to Azure AD DS and managed by using Group Policy. You have the ability to create organizational units to organize the VMs.

Note: Azure AD DS is a separate domain and not directly linked to your on-premises AD DS.

To simplify user access to resources joined to Azure AD DS, users credentials are the same as those in on-premises AD DS. The UPN and password are the same in both environments. The user accounts are synchronized as follows:

  • On-premises AD DS --> Azure AD Connect
  • Azure AD Connect --> Azure AD
  • Azure AD --> Azure AD DS

Implementing Azure AD DS avoids the need to create a VPN from on-premises to Azure to support hosting a domain controller in Azure. It also avoids the need to manage domain controllers in Azure.

More information:

Tuesday, February 9, 2021

OAuth Certificates with Hybrid Exchange

Older versions of Microsoft Exchange in a hybrid configuration with Exchange Online (EXO) used a federation trust to authenticate connections for free/busy information. Newer hybrid deployments of Exchange 2016/2019 use OAuth authentication instead of federation.

OAuth authentication is reliant on the Auth certificate in your on-premises Exchange. This certificate is created automatically with a lifetime of 5 years when you install Exchange Server on-premises. If this certificate has been replaced, then you also need to update Azure AD with the new certificate information. The simplest way to update the information is by running the hybrid wizard again after you update the Auth certificate.

I wrote a previous post about renewing/updating the Exchange Server Auth certificate here:

If you update the Exchange Server Auth certificate and forget to update the information in Azure AD, you are likely to see free/busy lookups to EXO fail. I recently saw this as a client and decided to dig into the configuration a little bit more.

Testing OAuth Connectivity

You can test OAuth authentication from the Exchange Management Shell on-premises. When doing this, you need specify a local mailbox with the following command:

Test-OAuthConnectivity -Service EWS -TargetUri 'https://outlook.office365.com/ews/exchange.asmx' -Mailbox user@domain.com -Verbose | FL

With the FL in the above command you'll see detailed information returned. If you run the command without FL and it's successful, you'll see output like this:


If the test is unsuccessful, you'll see text something like:

  • The remote server returned an error: (401) Unauthorized
  • Unable to get the token from Auth Server
  • Client assertation contains an invalid signature [Reason - The key was not found]

Verifying the Certificate Used for OAuth

To identify the Auth certificate currently used by on-premises Exchange Server, run Get-AuthConfig. In the example below, you can see the thumbprint for the currently used certificate. If the Auth certificate had been updated, it may show a previous certificate thumbprint too. The service name returned by this command is a GUID for Exchange Online.

Once you have the thumbprint of the Auth certificate, you can verify it exists on each of the on-premises Exchange servers. All servers should have the same Auth certificate. Use the following command for each of your on-premises Exchange servers.

Get-ExchangeCertificate ThumbprintFromAuthConfig -Server ServerName

Verifying Certificate Information in Azure AD

The Auth certificate configured in on-premises Exchange Server is used for client authentication to Azure AD for free/busy lookups. The public portion of the certificate is stored in Azure AD for this purpose. You can view this information by using the MSOL or AzureAD cmdlets.

To view the certificate information with the MSOL cmdlets, run the following command:

Get-MsolServicePrincipalCredential -ServicePrincipalName "00000002-0000-0ff1-ce00-000000000000" -ReturnKeyValues $true


This command may return multiple results. In my test environment, there are multiple entries for the same certificate. I assume that this is because it gets added each time I run the hybrid wizard.

The easiest way to verify that the certificate in Azure AD matches your on-premises Auth certificate is by using the StartDate and EndDate fields. These will match the NotBefore and NotAfter properties shown by Get-ExchangeCertificate. Be aware that StartDate and EndDate are in UTC time and the NotBefore and NotAfter are probably showing in your local time zone.

You can also save the Value property in a text file with the extension .cer. Then you can open the .cer file and view the certificate information. This includes additional information such as the Thumbprint which you can verify against the on-premises Auth certificate.


You can use the Azure AD cmdlets to get similar information (but not the certificate value). This is a bit more complex and requires multiple steps.

To get the ObjectID of the SPN for Exchange online:

$spnID = (Get-AzureADServicePrincipal | Where DisplayName -eq 'Office 365 Exchange Online').ObjectID


To list the certificates that have been uploaded:

$certs = Get-AzureADServicePrincipalKeyCredential -ObjectID $spnID


To get the thumbprint of the most recent certificate uploaded:



 Update Auth Certificate in Azure AD

If you find that the on-premises auth certificate is not present in Azure AD, the best solution is running the hybrid wizard again. Running the hybrid wizard will update the OAuth certificate.

If for some reason you don't want to run the hybrid wizard, you can update the certificate manually by exporting it from on-premises and importing it into Azure AD.

Steps 3 and 4 in the following document describe how to manually export and import the Auth certificate:

Links found during research