Windows Server 101 (Free eBook)

I just saw that a colleague Brian Svidergol (@bsvidergol) has released a free ebook, named Windows Server 101, about the basics of Windows Server. I've worked with Brian on several projects and he knows his stuff.

If you are new to working with Windows Server or want to brush up on the basics, I suggest you check it out:

• https://t.co/OC4ciZYVE9 (redirect through twitter)
• https://dl.orangedox.com/onKwZSVjAbbFFzBfs7 (direct link)

Certificate Template Versions and Autoenrollment

Certificate templates for Active Directory Certificate Services (AD CS) have multiple values related to versioning. In the Certificate Templates console, you can see two versioning attributes:

• Schema Version - This defines the options available in a Certificate Template. If you search for information about certificate template versions (such as https://technet.microsoft.com/en-us/library/cc725838(v=ws.11).aspx), the reference to different versions is the schema version. These schema versions are consisten across Windows servers.
• Version - This number is unique for your AD CS implementation. When you modify the template, this version number is incremented.

Byron Web Server template: Schema Version 2, Version 100.3

The version number for your certificate templates is composed of a major version number and a minor version number. In this example:

• Major version: 100
• Minor version: 3

When you make any edit to a certificate template, the minor version number is incremented. Even minor edits such as changing the security configuration for the certificate template increment the minor version number. This number is primarily for your own auditing purposes to identify that a change has been made. Incrementing the minor version number has no immediate impact on clients using autoenrollment.

When you right-click a certificate template and select Reenroll All Certificate Holders, the major version number is incremented and minor version number is reset to zero. Clients using autoenrollment see that major version has been incremented and renew their certificate using the updated certificate template.

If you use ADSIedit to view the properties of a certificate template, you can see the major and minor version numbers stored as the following attributes:

• revision
• msPKI-Template-Minor-Revision

If you manually edit the revision attribute and increment the value, it will trigger an update for autoenrollment clients just as if you had selected the Reenroll All Certificate Holders option in the Certificate Templates Console.

On the client side, autoenrollment is triggered by a scheduled tasks in \Task Scheduler Library\Microsoft\Windows\CertificateServiceClient. The triggers for enrollement are:

• SystemTask (for computer certificates): At startup, repeat every 8 hours
• UserTask (for user certificates): At sign in, repeat every 8 hours

If you are testing, you can manually run these tasks rather than restarting the computer or signing out and signing back in.

Unable to Start Data Collector Set

I was wanting to collect some performance information on a Windows 2008 R2 domain controller. One of the things you may not realize is that Windows Server includes some predefined data collector sets for common tasks and generates reports based on the data. Since I was having a performance issue on the DC, I wanted to run the Active Directory Diagnostics data collector set.

Built-in data collector sets

When I attempted to start the Active Directory Diagnostics data collector set (or any of the other predefined data collector sets), they didn't start. There was no error message or any indication of what the error might be.

Ultimately my workaround was to create a new data collector set with the same settings. Fortunately, when you create a new data collector set, one of the options is to create from a list of templates, which includes Active Directory Diagnostics. The new data collector set ran without any issues.

Selecting a template for a data collector set

How you do things matters (PowerShell Performance)

I've run into a couple of things over the last month or so that were interesting from a PowerShell perspective. With relatively small amounts of data, performance really isn't a concern. If you're only working with a few hundred or few thousand items PowerShell is pretty quick no matter what. However, when you work with larger datasets, how you do things can make a big difference.

Reducing the number of operations

The first thing I want to highlight is that that the number of operations affects performance. I was recently building some scripts to build a test environment by copying the AD structure from production. I based all of my work on AD Mirror (https://gallery.technet.microsoft.com/scriptcenter/AD-Mirror-PowerShell-Module-331b1b12).

I ran into a performance issue populating group membership. The code from AD Mirror had the list of group members in a variable and added them by using a foreach loop like this:

foreach ($m in $newMembers) {
     Add-ADGroupMember -Identity $targetgroup -Members $m
}

This worked, but for groups with a large membership it was very slow. A group with 10,000 members took about 10 minutes to complete.

After doing some reading, I realized that the -Members parameter will accept an array. So, I changed it up to look like this:

Add-ADGroupMember -Identity $targetgroup -Members $newMembers

It still wasn't fast, but it was a couple of minutes instead of 10. It's still slow, but when you multiply that out across the 18,000 groups I needed to get setup, it added up to a lot of saved time.

As a side note, this is still slow because there is a lot to update in AD when the change is made. When a group member is added, both the group and user accounts need to be updated. Adding the members as a single lump means only a single change to the group, but each user account is still updated. For a group with 10,000 members, this means the loop does 20,000 updates while adding as a single set does 10,001 updates.

However, I did run into a problem with the largest groups. There must be some sort of limit on the data that can be passed or a timeout, because I had two groups with 27,000 and 20,000 users that didn't work by using the full array. Initally I went back to doing those with a loop. However, then I thought that a better way would be splitting the array into smaller sets and do them as several groups.

In my case, the sizing parameters are pretty well defined. So, I'm able to break the array up based on defined size limits. I'm sure there is a way to more gracefully break this up using variables instead of hard coded limits, but this met my needs for the moment.

This is my code that breaks up the array into chunks of 10000 accounts:

$newMembersCount=$newMembers.count
$part1 = $newMembers[0..9999]
Add-ADGroupMember -Identity $TargetGroupDn -Members $part1
   
Switch ($newMembersCount) {
   { $_ -gt 10000 } {
       Write-Host "Second 10K users"
       $part2 = $newMembers[10000..19999]
       Add-ADGroupMember -Identity $TargetGroupDn -Members $part2
       }
   { $_ -gt 20000 } {
       Write-Host "Third 10K users"
       $part3 = $newMembers[20000..29999]
       Add-ADGroupMember -Identity $TargetGroupDn -Members $part3
       }       
   { $_ -gt 30000 } {
       Write-Host "Fourth 10K users"
       $part4 = $newMembers[30000..39999]
       Add-ADGroupMember -Identity $TargetGroupDn -Members $part4
       }
} #End switch

Lists instead of arrays

When you want an array of objects in PowerShell, you typically start it by declaring an array like this:

$arrayvar = @()

Then you add stuff to it like this:

$arrayvar += $newitem

When you create and manage an array this way it ends up being very inefficient for large arrays. The array you create is a fixed size. Then when you use += to add an item, PowerShell creates a new array that contains the existing values and the new value. In a very large array that ends up being a lot of data movement with no benefit.

The fix for this is to create an array as a list that is not a fixed size:

$listvar = New-Object System.Collections.ArrayList

Then add items to the list like this:

$list.add($newitem)

The add function for a list is many times faster than using += for a fixed size array. When you are working with large lists it becomes exponentially faster. It's hard to quantify because you'll see varying performance differences based on the objects you're working with. However, this link has one example:

• https://learn-powershell.net/2014/09/21/quick-hits-adding-items-to-an-array-and-a-look-at-performance/

Here is some data from that link:

• 10,000 items, array performance: 16 seconds
• 10,000 items, list performance: 1 second
• 100,000 items, array performance: 3230 seconds (53 minutes)
• 100,000 items, list performance: 7 seconds

Update Apr 2017
The original version of this post suggested to use generic list instead of an array list. Someone with much more technical knowledge on this than me has suggested that array list is a better way to go. So, I've change the syntax above to use array list instead of generic list.

SCOM AD Monitoring Alerts

I've been working with a larger client for the last several months on Active Directory (AD) issues. One of the ongoing small issues has been AD monitoring alerts generated in System Center Operations Manager (SCOM) when it appears nothing is actually wrong.

The alerts were appearing intermittently on several of the servers, but not all. We were seeing alerts like this:

Failed to ping or bind to the Infrastructure Master FSMO role holder
AD Op Master Response : The script 'AD Op Master Response' could not determine the schema Op Master.The error returned was: 'LDAP://DC.contoso.com/RootDSE' (0x8007203A)

Failed to ping or bind to the Schema Master FSMO role holder
AD Op Master Response : The script 'AD Op Master Response' could not determine the schema Op Master.The error returned was: 'LDAP://DC.contoso.com/RootDSE' (0x8007203A)

Failed to ping or bind to the RID Master FSMO role holder
AD Op Master Response : The script 'AD Op Master Response' could not determine the RID Op Master.The error returned was: 'LDAP://DC.contoso.com/RootDSE' (0x8007203A)

Failed to ping or bind to the PDC FSMO role holder
AD Op Master Response : The script 'AD Op Master Response' could not determine the PDC Op Master.The error returned was: 'LDAP://DC.contoso.com/RootDSE' (0x8007203A)

Failed to ping or bind to the Domain Naming Master FSMO role holder
AD Op Master Response : The script 'AD Op Master Response' could not determine the domain naming Op Master.The error returned was: 'LDAP://DC.contoso.com/RootDSE' (0x8007203A)

Script Based Test Failed to Complete
AD General Response : While running 'AD General Response' the following consecutive errors were encountered: Failed to bind to 'LDAP://DC.contoso.com/rootDSE'. This is an unexpected error. The error returned was 'LDAP://DC.contoso.com/rootDSE' (0x8007203A)

We also implemented AD Client monitoring and it showed the following errors:

Script Based Test Failed to Complete
AD Client Connectivity : The script 'AD Client Connectivity' failed while getting 'LDAP://DC.contoso.com/RootDSE.The error returned was: 'LDAP://DC.contoso.com/RootDSE' (0x8007203A)

Script Based Test Failed to Complete
AD Client Monitoring: AD Connectivity is unavailable, or the response is too slow. The bind to 'LDAP://DC.contoso.com/RootDSE' took 4259 milliseconds, which is longer than the allowed 1000 milliseconds.

After doing a packet sniff on one of the DCs, I was able to correlate a specific pattern of network activity with the failure to connect for AD Client Connectivity. When the connection error occurred the AD Client attempted to connect to the DC, but the DC refused the connection. In Network Monitor, this appears as:

• Client to server: TCP packet with the SYN flag set
• Server to client: TCP packet with Ack and Reset flags set

This behavior indicates that there is either no service listening on the port or that the service is refusing the connection. The service was definitely listening because it worked most of the time. So, the service must be refusing the connection for some reason. It wasn't a timeout of any type because the refusal was happening in about 1ms.

These domain controllers were running Windows Server 2008 R2. I did find some web pages, such as the one below, that indicated this type of error can be caused by IPv6 being disabled in Windows Server 2012:

• https://blogs.msdn.microsoft.com/nicole_welch/2015/02/monitoring-window-2012-ad-multiple-monitoring-scripts-failing-or-why-i-need-ipv6/

I can confirm that enabling IPv6 also fixed this issue for Windows Server 2008 R2. However, it didn't fix the problem immediately. If we had restarted the DCs, it probably would have taken effect immediately, but we enabled IPv6 during production hours and couldn't reboot right away. About 7 hours after IPv6 was enabled, all alerts related to the AD Client monitoring stopped and it's been all good since.

In our scenario, it was a little more difficult to be confident this would work because the only two servers experiencing this issue regularly were in a single location. That led us to think that there might be something odd in the network configuration there. Both of those servers were configured to use WINS for some legacy stuff in that site. I think that might have been an influencing factor somehow as that's the only configuration difference I could find.

Update:
The two servers having these errors had slow LDAP client bind times even after these errors went away. After a lot of monitoring, we added additional virtual processor to those two DCs. We increased from 2 vCPU to 4 vCPU. Once that was done, the slow LDAP client bind times disappeared. So, insufficient processing power seemed to play a part in this also. There were two DCs servicing a very large site (I think over 5000).

Office 365 vs. On-Premises Exchange Server

A large client is currently running Exchange 2010 and is evaluating moving to Office 365 vs upgrading to Exchange Server 2016. I talked with them about it and thought it would be useful to document it for future reference.

If you are a very small organization, then Office 365 is a slam dunk. It's going to perform better and be more cost effective than your could ever implement on your own. This is even before we consider the cost of the the consultants to get your on-premises Exchange up and running.

For mid-sized and large businesses there are more things to think about....

Cost

Direct cost is the first thing everyone wants to evaluate when considering Office 365. Your exact costs are going to vary depending on how you want to implement Exchange and which Office 365 plans you think are appropriate. So, I'm going to let you figure out the exact costs, but here are the things you need to consider:

On premises:

• Exchange Server licenses
• Exchange Server CALs (basic and enterprise if required)
• Hardware for the Exchange servers
• Storage for the Exchange servers
• Backup software and storage
• Antispam and antivirus solution

 Office 365:

• Monthly or yearly license (subscription-based)

High Availability

On-premises Exchange Server can be made highly available. This is done by configuring multiple Exchange servers in a database availability group (DAG) and by using load balancers. This configuration is well understood but definitely requires some planning and expertise to implement.  In most cases, hardware load balancers are used which are an additional cost.

Other high availability considerations for on-premises Exchange include redundant Internet connections, and alternate data centers. To provide a solution that is truly highly available, it takes a lot of redundant infrastructure.

Office 365 is highly available out of the box. You don't need to do anything, that's just the way it's designed. Your data is automatically located in multiple data centers in your region. If one data center fails, your data is still safe in another data center and you're still able to access it. This is one of the big advantages for Office 365, particularly for mid-sized organizations that might not have the resources to deploy office 365 across multiple data centers.

Authentication

On-premises Exchange Server uses Active Directory (AD) for authentication. When users are on a domain joined computer, authentication is performed automatically by Outlook. Externally, the AD username and password are used.

For Office 365, there are multiple ways that authentication can be configured. First lets talk about user accounts...

Very small organizations might choose to create user accounts in Office 365 manually. For these organizations, the users will sign in by using their email address and a password that is set in Office 365. This password could be different from the AD password on-premises and there is no synchronization of the passwords between AD and Office 365.

Larger organizations will use Azure AD Connect to synchronize user accounts with Office 365. This avoids the need to manually create the accounts. Azure AD Connect also has an option to synchronize passwords with Office 365. This simplifies password management for users. It also simplifies password resets when users call the help desk.

When passwords are synchronized with Office 365, all authentication is still handled by Office 365. There is no reliance on any on-premises infrastructure for the authentication process. So an on-premises service outage does not cause an Office 365 service outage.

Very large organizations may consider using Active Directory Federation Services (AD FS) for Office 365 authentication instead of password synchronization. When AD FS is implemented, Office 365 directs authentication requests to AD FS which then authenticates the request against AD. This is significantly more complex to configure than password synchronization but provides the following benefits:

• True single sign-on where workstation credentials are passed through for authentication just like on-premises Exchange.
• When an on-premises AD account is locked or disabled, this also applies to Office 365 authentication. Without this, disabling an Office 365 account is a separate process.

A key point to remember is that if your AD FS infrastructure is down, so is Office 365 authentication.

Account Lockout

For on-premises Exchange, the account lockout policies in AD are applied to authentication for Exchange mailboxes. The only exception is if a device such as a proxy is performing pre-authentication. If pre-authentication is performed then the device providing pre-authentication might have more restrictive policies than what is configured in AD.

When password synchronization is used with Office 365, there is a account lockout mechanism included in Office 365. After 10 failed attempts, an account is locked for 1 minute and if account lockout is triggered multiple times the lockout time is increased each time. This account lockout mechanism is not configurable.

When AD FS is used with Office 365, the account lockout policies in AD are used.

Mailbox and Archive Size

In on-premises Exchange, the maximum size of mailboxes and archive mailboxes is limited by the storage design. Or, more often by the cost of the storage being used. Using direct attached storage (DAS) with 7.2K SAS drives (rather than a SAN) is the most cost effective solution to provide large mailboxes and archives.

In Office 365, the maximum size of mailboxes and archive mailboxes is limited by the license assigned to the mailbox:

• Kiosk: 2 GB mailbox, no archive
• E1: 50 GB mailbox, 50 GB archive
• E3: 50 GB mailbox, unlimited archive
• E5: 50 GB mailbox, unlimited archive

Online Reading and Editing Attachments

In Exchange 2010, there was basic functionality for rendering attachments as web content in OWA. In Exchange 2016, this functionality is not included. Instead, you need to implement Office Online Server. Office Online Server is included with most volume licensing agreements for Exchange Server 2016.

Implementing Office Online Server is not terribly complex for a single server, but requires at least an additional virtual machine. If you want the functionality to be highly available you need to implement multiple servers and load balancing.

Office 365 includes the ability to read and edit attachments by default. No additional configuration is required.

Spam and Antivirus Filtering

For on-premises Exchange, you need to purchase additional products for spam and anti-virus filtering.

Office 365 licenses include Exchange Online Protection for anti-spam and virus protection at no additional cost.

Network Configuration

Most implementations of on-premises Exchange Server require you to implement proxies, load balancers, and firewalls. External access from the Internet to the Exchange servers also need to be configured.

For Office 365, you just need to ensure that clients on the internal network can communicate with Office 365 over the Internet.

Client Communication

Most on-premises deployments of Exchange server use Outlook in cached mode. However, because the internal network is reliable and fast, you can implement Outlook in online mode instead. This can be useful if you are concerned about the size of cached data on computers that are shared by multiple users.

Outlook clients must communicate with Office 365 servers over the Internet. This may result in higher Internet bandwidth usage. Generally, Outlook cached mode is used to minimize network utilization and provide better performance over networks with high latency.

Recent versions of Outlook cache only recent data in a mailbox rather than the entire mailbox. This mitigates concerns about large mailboxes being cached in local profiles. The amount of data cached is configurable.

Migration

Exchange Server 2016 can co-exist with Exchange Server 2010 or Exchange Server 2013. Migration is just a matter of moving mailboxes to Exchange Server 2016. There is almost no impact on users and the Outlook client is automatically redirected to the new mailbox location.

Office 365 can be integrated with Exchange Server 2010 or later by using hybrid mode. Once in hybrid mode, mailboxes can be migrated to Office 365 with minimal impact on users. Outlook is automatically reconfigured when mailboxes are moved.

Backup

Most common backup software can be used for backup with an appropriate agent on the Exchange servers. The restore functionality in the software varies but most can restore individual mailboxes or databases. This software is an extra cost.

There are third party options available for backing up Office 365 mailboxes, but there is no built-in functionality in Office 365 for this purpose.  However the same Single Item Recovery available in on-premises Exchange can be used in Office 365 to recover items for up to 30 days (default 14 days) after they are removed from deleted items.

Microsoft maintains multiple copies of mailbox data to avoid the need to restore from backup due to server error.

If you use E3 licenses, you can implement a hold on mailboxes to permanently retain messages received or sent from mailboxes. This held data is searchable for discovery purposes. Data holds can also be implemented for specific retention periods. Microsoft is positioning this type of hold to replace journaling and archiving solutions.

Management

For an on-premises Exchange deployment you need to perform the following management tasks:

• Monitor performance
• Monitor hardware for failures
• Update firmware
• Apply operating system updates
• Apply Exchange Server updates
• Manage mailboxes

On a system that is properly designed, there is minimal need to monitor performance (you built it with some head room), but there is an ongoing need to address issues such as failed indexes and large delivery queues. These issues are not relevant for Office 365 management.

As a cloud-based service, you are not responsible for any monitoring or updates in Office 365. Management of user mailboxes is still required.

New management tasks for Office 365 include:

• Managing directory synchronization
• Managing user licenses

Directory synchronization typically requires little management. There may be occasional troubleshooting, but it is rare. Generally once directory synchronization is configured, it just works.

Processes for creating new mailboxes do need to be modified to include assignment of Office 365 licenses. This can be scripted and run as a scheduled task if it is not required to be time sensitive.

If all users are migrated to Office 365 a local Exchange server is still maintained for management purposes. This is required because changes are made in the local AD and synchronized to Office 365. However, outages for this management server do not affect Office 365 users only management functionality. Microsoft provides a free license for this purpose.

Additional Features

Many Office 365 deployments begin as a replacement for on-premises Exchange. However, for mid-sized and large organizations, when you do a raw comparison of Exchange on-premises costs and Office 365 licensing costs, you may find that Exchange on-premises is actually a lower cost.

You need to consider that Office 365 includes more than just Exchange functionality. Depending on the licenses you select, there are various features. Also note, that you can mix and match licenses as required to give different features to different users to meet the needs of your organization.

For an accurate list of Office 365 plan features, see the Microsoft web site. However, here's a quick summary you can use to get a general idea of what's included with each license type:

• Kiosk:
• Exchange Online
• SharePoint Online
• E1: Kiosk features plus
• One Drive for Business (1TB)
• Skype for Business
• Yammer
• E3: E1 features plus
• Office 365 Proplus (desktop office suite)
• Azure Rights Management
• E5: E3 features plus
• Cloud PBX

These extra features provide more value than just Exchange online. I've been surprised at the number of larger organizations that have subscribed to Office 365 with E3 licenses as their preferred method of deploying Microsoft Office. Skype for Business is also a bit of a pain to implement on-premises and having that included with the E1 licenses is beneficial.