Microsoft Exchange Search Host Controller Failing

Got a call from a client that reported user searches in OWA and Outlook were failing. This particular client had recently has some severe SAN issues. So, who knows were the problem could be. However, the place to start is with the indexes, and the index on this server showed as healthy.

Next, I looked at the services. Microsoft Exchange Search was running properly, but Microsoft Exchange Search Host Controller was stopped. When I started Microsoft Exchange Search Host Controller, it showed as running for a few seconds and then stopped.

In the event log, two related events were showing:

Event ID: 1026
Source: .NET Runtime
Application: hostcontrollerservice.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.Ceres.HostController.Controller.HostControllerException
Stack:
   at Microsoft.Ceres.HostController.WcfServer.WcfService.StartService()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart()

Event ID: 1000
Source: Application Error Faulting application name: hostcontrollerservice.exe, version: 15.0.4454.1006, time stamp: 0x50d08ef5 Faulting module name: KERNELBASE.dll, version: 6.2.9200.16857, time stamp: 0x530e784b
Exception code: 0xe0434352
Fault offset: 0x0000000000047b8c
Faulting process id: 0x413c
Faulting application start time: 0x01cf86480e45a787
Faulting application path: C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\hostcontrollerservice.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 4cc1a310-f23b-11e3-943e-00155d09980c
Faulting package full name:
Faulting package-relative application ID:

I found a couple of references to a corrupt and empty ini file that can cause this issue. The file in question is C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data\Nodes\Fsis\node.ini. However, mine appeared to be good. There was content in the file in any case.

My fix ended up being removing and reinstalling the search services. This was based on advice given to another person by Microsoft support. The steps were:

1. Ensure that both Microsoft Exchange Search and Microsoft Exchange Search Host Controller are stopped. I had to manually kill noderunner.exe processes that were started by Microsoft Exchange Search Host Controller but not stopped when the service failed.
2. Create a backup copy of InstallConfig.ps1 in C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\Installer.
3. Edit InstallConfig.ps1 and modify the line for the baseport variable to be $script:baseport:3800. On this system the original value was 17000. I'm not sure if that's consistent or varies across systems.
4. Uninstall search by running: .\InstallConfig.ps1 -u -datafolder "C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"
5. Reinstall search by running:   .\InstallConfig.ps1 -i -datafolder "C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"

Other links for research into this issue:

• http://social.technet.microsoft.com/Forums/exchange/en-US/74aef23c-fe67-4f2f-a74e-4e895db28fa9/microsoft-exchange-search-host-controller-service-stopped-no-update-installed?forum=exchangesvrgeneral
• http://plasmasphere.co.uk/exchange-2013/exchange-2013-hostcontrollerservice-exe-crashes/

Estimator for Disk IOPS

One of the things I've always had issues with is finding a good way to estimate disk IOPS when I'm planning out systems. This is particularly important now with Hyper-V hosts and multiple VMs.

Today by luck, I saw someone in a forum talking about web page to provide estimated IOPS. I took a look and I love it.

This calculator lets you enter in the type of drive, hard drive size, number of disks, read percentage, and write percentage. Based this it gives you estimated IOPS for RAID 5,6,and 10.

The tools is here:

• http://www.thecloudcalculator.com/calculators/disk-raid-and-iops.html

Adding Favorites to Visual Studio Explorer

When I do course writing projects for Microsoft, we use Source Control Explorer from Visual Studio to control access to the documents. You may also know this as Team Foundation Studio (TFS).

I recently upgraded my laptop and at the same time downloaded the version of Source Control Explorer for Visual Studio 2013. Previous versions would start at the same point in the file structure as where I left off. But VS 2013 likes to collapse the entire folder structure and forces me to browse through the whole structure to get to my work.

Fortunately, I've just found a free add on for Visual Studio that lets you create favorites. I'm not using this to get me where I need to be in the folder structure more quickly. It's a minor thing, but it saves me a few minutes each time I open it.

To learn more or obtain this extension see the developer's web page here:

• http://vlasovstudio.com/favorite-documents/

Also, for Visual Studio 2013, it does not work unless you also install the Microsoft Visual Studio 2013 Shell (Isolated) software. If this is missing, the extension doesn't load properly. This is noted by the developer, but it's easy to miss. You can obtain the necessary software here:

• http://www.microsoft.com/en-ca/download/details.aspx?id=40764

TechEd 2014 Sessions Available at Channel 9

In case you (like me) did not have time to make it down to TechEd 2014, they've posted recorded sessions on Channel 9. It's pretty much like being there.

• http://channel9.msdn.com/Events/TechEd/NorthAmerica/2014

Forward Messages and Leave in Mailbox

One of the fairly common tasks for Exchange administrators is to configure forwarding on a mailbox. Previous versions of Exchange gave you the option to leave a copy of forwarded messages in the mailbox and the option was accessible in the graphical administration tools. As you can see in the screenshot below, for some reason in Exchange 2013 this option was left out of the graphical/web interface.

When you enable forwarding in the web interface, by default the message will not be left in the recipient mailbox. However, the option is still there if you use PowerShell.

The Set-Mailbox cmdlet is your friend:

Set-Mailbox -Identity "user" -ForwardingAddress "otheruser@domain.com" -DeliverToMailboxAndForward $true

Microsoft Exchange Conference 2014 - Sessions Online

I was too busy to attend the Microsoft Exchange Conference this year in Austin. Fortunately for us they recorded most (if not all) of the breakout sessions and made them available on Channel 9. It looks like over 50 session are online and available for free.

You can see them here:

• http://channel9.msdn.com/events/mec/2014

New Pagefile Sizing for Exchange 2013

Over the last few years, memory has become increasingly inexpensive. As a result, for scalability, you can often see Exchange servers with 64GB or 128GB of RAM.

In the past, Microsoft's guidance on pagefile size has been system memory plus some amount to allow for the contents of memory to be written to disk in the event of a crash. Basically, this is for troubleshooting information. For Exchange 2013, the guidance has been RAM+10MB.

Unfortunately this means that a large amount of disk space is used for that pagefile with little benefit because it's not typically used much. Certainly not the full RAM of the system. I've been arguing for a long time that unless you are troubleshooting bluescreen events with Microsoft support, there is no need for this large of a pagefile for Exchange or regular Windows servers.

As of Exchange 2013 SP1 the new guidance is to max out the pagefile at 32GB+10MB. Apparently Microsoft has been doing this internally for some time.

For more information:

• http://blogs.technet.com/b/exchange/archive/2014/04/03/ask-the-perf-guy-sizing-guidance-updates-for-exchange-2013-sp1.aspx