Monday, March 28, 2016

DPM Replica Is Inconsistent

I just completed a new install of DPM for a client on a new server. The number of servers is fairly small. So, it wasn't worthwhile to migrate the existing backups to the new server. Instead, I just did a new install of DPM and reinstalled the DPM client software to point them at the new server.

After doing the initial backups, all looked good except for a generation 2 virtual machine running Windows Server 2012 R2. This server has the error Replica is inconsistent for Bare Metal Recovery and System State. All other data backed up fine.

Most of the problems I've ever had with DPM were due to Windows Backup not being installed on the server being backed up. Usually, installing that feature fixes any issues. However, in this case it was already installed. Which makes sense because the servers were being backed up successfully by the other DPM server.

The application event log gave me:
Event ID: 517, Error
The backup operation that started at DateAndTime has failed with the following error code '0x807800C5' (There was a failure in preparing the backup image of one of the volume in the backup set.)

The Windows\Backup event log gave me the same information plus some details in the XML view. One of the details was a path to the error log in C:\Windows\Logs\WindowsServerBackup.

Browsing to this location I found:
C:\Windows\Logs\WindowsServerBackup\Backup_Error-DateTime.txt
There were several instances of this file from running overnight. Each file had an error, but there were two possible errors.
Backup of volume \\?\Volume{GUID}\ has failed. The system cannot find the file specified.
Backup of volume \\?\Volume{GUID}\ has failed. The mounted backup volume is inaccessible. Please retry the operation.
This provided me with a volume GUID that I could use to track down the volume that was causing the issue. To see the volume GUIDs, I used the following PowerShell command:
Get-Volume | FL
From this information I could see that the volume causing the problem was a 300MB Recovery volume created automatically during installation. After doing some searching, I determined that the issue was caused by this volume having insufficient space to do a VSS snapshot during backup. In the 300 MB volume about 55MB was free.

There are three possible fixes for this issue:
  • Disable WinRE. When you disable WinRE by running reagentc.exe /disable then the backup does not create a snapshot of the recovery partition. It is backed up, but a snapshot is not used. This is fine if you want to leave the recovery partition disabled.
  • Move VSS storage to another drive. If you configure VSS to store snapshot changes to another volume such as C: then the backup completes properly. This is simple and effective. It takes little space on the other volume. So, there is no negative impact on storage space. The required command: vssadmin add shadowstorage /for=\\?\Volume{GUID}\ /on=c: /maxsize=500MB
  • Create larger volume for WinRE. You can create a new larger volume for WinRE and reinstall the WinRE files on that partition. While this initially seems like the graceful choice, it is the hardest to implement. You need to have sufficient unallocated disk space. I personally wouldn't recommend it.
Additional resources:

Thursday, March 24, 2016

Windows 2008 R2 Install Hangs on Dell T330

Had an interesting issue on a new Dell server this week. Normally we are installing Windows Server 2012 R2 but a client had a requirement for Windows 2008 R2 on this particular server to run DPM 2010. Did our usual installation route, but the install hung at the first screen where you select language and keyboard settings. Initially I thought this might be occurring because I had the ISO mounted remotely through the iDRAC, but the same error occured on site too.

It turns out that the problem was due to USB 3.0 drivers. The newest Dell servers have only USB 3.0 ports and Windows Server 2008 R2 does not include USB 3.0 drivers, only USB 2.0 drivers. For the higher end R430/T430 servers you can change the USB support to 2.0 in the BIOS for installing older operating systems. For the R330/T330 and lower, it isn't possible to turn off USB 3.0 in the BIOS.

Ultimately the solution was to inject USB 3.0 drivers into the Windows Server 2008 R2 install media. Which sounds like a giant pain but Dell provides a utility that does it for you. The Driver Update Utility adds the USB 3.0 drivers and if you have other drivers, you have an option to add them also.
UPDATE: Some of the newer Dell models have an option in the firmware to disable USB 3.0 and use USB 2.0. Check that out if your having this issue. Be aware that the less expensive servers do not have this option. However, I don't have a list of which ones do and don't have the feature.

Thursday, March 17, 2016

Block Messages Spoofing Your Domain

Spam with attached malware has been going crazy lately. It's a complaint that all of our clients are dealing with. Antispam software doesn't seem to be able to keep up with new variants that are showing up each day.

Many of the spam messages spoof your domain in the sender address. For example, I might receive a spam message from admin@conexion.ca when my email address is in the conexion.ca domain. Fortunately, in Exchange Server, we can modify the Internet receive connector to block messages from your accepted domains. This means that emails with your domains as the From address will not be accepted from the Internet.

To block inbound messages from our own domain, we remove the extended AD permission ms-exch-smtp-accept-authoritative-domain-sender for the anonymous user on the Internet receive connector. Assuming that have already identified that connector, the following command removes the permission:
Get-ReceiveConnector "Internet" | Get-ADPermission -User "NT Authority\Anonymous Logon" | Where {$_.ExtendedRights -like "ms-exch-smtp-accept-authoritative-domain-sender"} | Remove-ADPermission
If you're like me and want to verify that you've got the right thing before you remove it, use the following command to verify first:
Get-ReceiveConnector "Internet" | Get-ADPermission -User "NT Authority\Anonymous Logon" | Where {$_.ExtendedRights -like "ms-exch-smtp-accept-authoritative-domain-sender"} | Format-List *
Before you actually implement this make sure that you don't have any applications or devices such as copiers that submit messages on the Internet connector:
  • I have seen copiers in small businesses configured to submit scanned documents via email through the SMTP relay of an ISP instead of the local Exchange server.
  • I have also seen external cloud-based applications sending messages to internal users using a from email address of the internal domain.
  • Also, sometimes Internal devices are configured with an IP address that uses the same receive connector as used for Internet mail reception. So, verify that the receive connector used for Internet messages does not allow internal IP addresses.
This permission exists for Exchange 2010, Exchange 2013, and Exchange 2016.

Wednesday, March 16, 2016

Missing "How will this person sign in?"

Normally when you create a user account in Windows 10, the first thing asked is How will this person sign in?" This identifies whether  the account is a local account or a Microsoft account. Today as I was developing some course material for user accounts in Windows 10, this screen was missing from my laptop. Instead, it went directly in to asking for account information for a local user account.

After much searching around, I finally figured out that my laptop had no network connectivity. I had just installed a Windows Insider build and the new build killed WiFi on my laptop. Once the WiFi connectivity was restored the new account wizard behaved as expected.

I expect you would see the same behaviour in Windows 8.1 when disconnected from the network.

Tuesday, March 15, 2016

No Quotes Required for Paths with Spaces

I got a comment back from a technical editor today that I didn't need quotes around the path in a CD command at the command prompt. I was surprised because the path included spaces. However, after testing, it's true!

In the past I would have done:
cd "folder1\foldername with spaces"
However, this version (without quotes) works just as well:
cd folder1\foldername with spaces
Since it worked in Windows 10, I tried out Windows 8.1. It works in Windows 8.1 too. It's funny the little things you learn sometimes.

Friday, March 11, 2016

Outlook Slow in Online Mode

Ran into an odd issue with Outlook running slow when in online mode and accessing a mailbox on Exchange Server 2013. Not actually my client, but trying to help out a colleague. Also, not actually my fix, my colleague Patrick figured it out.

The scenario:
  • Outlook 2007 clients with minimum updates to work with Exchange 2013
  • Exchange 2013 CU4 (SP1)
  • Legacy software that makes upgrading client and server problematic
  • Exchange 2013 running on physical server works fine.
  • Migrated mailboxes to DAG and CAS running as VMWare VMs and Outlook is slow.
As there was a lot of legacy stuff in this environment, every good computer geek likes to blame old stuff. However, that didn't change the fact it all worked fine on the physical server, but not the virtual ones. So, the overall configuration while not optimal, should have been functional.

We looked at standard stuff like misconfigured URLs, certificate configuration, antivirus configuration, resources allocated to the VMs. None of it seemed to be an issue. We changed a few things, but what we did generated no improvement.

Finally, my colleague found some information about Exchange 2013 and recent versions of VMWare Tools causing issues with network connectivity. I've had issues in the past where we needed to update to the latest tools, but nothing like this.

The solution (for now) ended up being removal of the VMWare Tools from the VM. Based on research, this appears to be related to TCPAckFrequency. I don't normally implement on VMWare, but I'll be watching for this in the future.

It's also worth noting that this would not have been an issue with cached mode. For this client, online mode was required.

Additional resources:
Update (Nov 2016)
There seems to be a general issue with VMware Tools version 9.4.10 (5.5u2) and later with Exchange Server. I've now run into this as a performance issue with various versions of Outlook and Exchange 2013/2016. What seems to be the best solution is downgrading the VMware Tools to version 9.4.5 (5.5u1).

Also notable is that it seems to be client dependent. It seems to be an issue with Windows 7 clients and either not at all or less so with Windows 8.1 or Windows 10 clients.

Thursday, March 3, 2016

More on Secure Time for Windows 10


After doing some experimenting, I have determined the following about secure time in Windows 10:
  • Secure time is updated when the computer goes to sleep or is shut down. This makes sense and I believe it is used to ensure that Windows has a reasonable time change and protect against irrational time sources. So, you want to compare time on startup against when it was last shut down.
  • The registry keys use UTC time. When I read the time in SecureTimeEstimated and apply the correct time zone, then it matches the time the computer was put to sleep or shut down.
  • SecureTimeEstimated seems to be the time of shutdown or going to sleep.
  • SecureTimeHigh is one hour later than SecureTimeEstimated.
  • SecureTimeLow is one hour earlier than SecureTimeEstimated.
To view these values yourself you can use the following PowerShell code:
[datetime]$(Get-ItemProperty HKLM:\System\CurrentControlSet\Services\W32Time\SecureTimeLimits).SecureTimeEstimated
[datetime]$(Get-ItemProperty HKLM:\System\CurrentControlSet\Services\W32Time\SecureTimeLimits).SecureTimeHigh
[datetime]$(Get-ItemProperty HKLM:\System\CurrentControlSet\Services\W32Time\SecureTimeLimits).SecureTimeLow
Please note that the times returned are 1600 years before the current date. So, even through the year may appear to be 0416, add 1600 to get the current year of 2016.

Previous info about secure time and the problems it can cause for imaging are here:


Tuesday, March 1, 2016

Windows 10 - Time Synchronization and Imaging

Windows 10 has a new feature for time synchronization called secure time. The problem is that I haven't been able to find any documentation on it. What secure time seems to do is prevent significant time changes that are considered unreasonable. I believe it's new because there is no documentation at this time and because the registry keys for it don't exist in Windows 8.1, but do in Windows 10.

Here is the problem I ran into with secure time.....

As part of course development, we have a base image of Windows 10 with Office 2016 installed. We use this base as a starting point for differencing drives in course VMs, but this would also apply to an image you deploy to new workstations. This issue occurred even with just the base drive deployed.

When I had a VM using the base drive, it would run fine for about half an hour and then the time would reset back to December 3rd. This is approximately when the base drive was created. Time in the VM would then bounce back and forth between the proper time it obtained from the PDC and December 3rd. If you reboot, it would use proper time again for about half an hour and then the same issue would recur.

If you're reading this post because you have VM time synchronization problems in general, you should also check out this posting for some detailed information about how time synchronization works with Hyper-V:
Also, see here for how to enable debug logging for time synchronization:
I tried several things described in these articles:
  • Manually configured time synchronization type to domhier.
  • Manually configured the PDC as time source.
  • Disabled the Hyper-V time synchronization in the guest OS of the VM.
  • Configured the PDC to use BIOS clock as a time source instead of time.windows.com (which was unreachable in a isolated environment).
All the time I was doing these things I had debug logging enabled and watched the log. Time synchronization would be done in 1024 second intervals (about 17 minutes). After two synchronization intervals time would reset to December 3rd. Finally, I really paid attention to the message in the debug log:
Setting the system time because it is outside the secure time limits.
Current system time: 18:0:45.622 3/1/2016
Target system time: 19:4:54.231 12/3/2015
The key here is the reference to secure time which I didn't realize existed. However, as I browsed the registry keys for time synchronization at HKLM:\System\CurrentControlSet\Services\W32Time (I'd been reviewing these over and over), it clicked and I noticed the registry key SecureTimeLimits. In SecureTimeLimits were the following:
  • SecureTimeConfidence
  • SecureTimeEstimated
  • SecureTimeHigh
  • SecureTimeLow
  • SecureTimeTickCount
The values for SecureTimeEstimated, SecureTimeHigh, and SecureTimeLow are all hex values. However, you can convert them to a date format. In my case, I used Windows Calculator to convert 0x1d12dfd7e79669e to decimal 130936430942316190. Then I used PowerShell to covert to a date:
[datetime]$STestimated=130936430942316190
$STestimated
This gave a date of December 3, 0415 7:04:54 PM. When time conversions are done, you need add 1600 to the year. So, this gave us December 3, 2015 7:04:54 PM. Exactly when the time was being reset to. Next question, how do we make it stop?

The key SecureTimeConfidence looks like a likely solution. The current value was 0x6. A quick Google search turned up only a single reference to this value:
Two other people in this thread indicated that setting this value to zero resolved the same issue for them. Unfortunately I've not seen any documentation on this registry value at all and I've been unable to find any. Another installation of Windows 10 Eval that I'm using as a demo has a value of 0x7. My laptop which is upgraded from Windows 7 has a value of 0x8.

However, the important point, is that I can confirm that setting SecureTimeConfidence to a value of 0 resolved the issue in the VM. I believe this will be a required part of preparing Windows 10 images in the future.

UPDATE: There is now a KB article related to this as provided by an anonymous comment below: https://support.microsoft.com/en-us/kb/3160312

ANOTHER UPDATE (Sept 2016): There is another article describing "Secure Time Seeding". This article also indicates that the problem described above in this posting has been fixed in Windows 10 Anniversary Edition. Check it out for more details: https://blogs.msdn.microsoft.com/w32time/2016/09/28/secure-time-seeding-improving-time-keeping-in-windows/